Reading files larger than 606666752 bytes

noobster

Member

Reaction score: 12
Messages: 92

I'm trying to read a large file in C using open() and read(). I am doing this is a non-blocking way using a kqueue. Everything works fine until 606666752 bytes have been read from the file descriptor, but at that point kevent stops returning (i.e., no bytes are ready to be read). For smaller files, this is not an issue. Is there any setting I have to change to be able to read large files? Thanks in advance.
 

Mel_Flynn

Well-Known Member

Reaction score: 81
Messages: 379

Are you testing for EV_ERROR or similar?

FreeBSD off_t should handle up to OFF_MAX, which is nowhere near the value you're seeing. Would be easier to see your code...
 
OP
OP
N

noobster

Member

Reaction score: 12
Messages: 92

All settings seem to be set to unlimited. Also, I'm sure I didn't edit this file.
 

Mel_Flynn

Well-Known Member

Reaction score: 81
Messages: 379

Let us know if this is the same problem as EOF, you may want to warnx("%lli", ke.data) just before going to the next loop sequence.
 
OP
OP
N

noobster

Member

Reaction score: 12
Messages: 92

Ah, it looks like that ke.data contains the value 606666752 in the first loop instead of the correct file size. Therefore, it only reads 606666752 bytes. I suspect this is because of the limited size of that struct field, but I have no idea how to get around this limitation. Anyone?
 

Mel_Flynn

Well-Known Member

Reaction score: 81
Messages: 379

When ke.data - BUFSIZ reaches <=0, issue EV_CLEAR on the next invocation? Not sure it'll work, but from coarse reading of the code, EV_CLEAR should clear the kernel copy of the event. Then you need to solve the problem to detect when really EOF.
 

Mel_Flynn

Well-Known Member

Reaction score: 81
Messages: 379

Is this file INT_MAX+606666752 bytes? Cause I can't reproduce it with a 640M file.
 
OP
OP
N

noobster

Member

Reaction score: 12
Messages: 92

The file I'm using is 4901634048 bytes, which is 4.6 gigabytes. So the file is much bigger than INT_MAX.
 
OP
OP
N

noobster

Member

Reaction score: 12
Messages: 92

It looks like the problem is with the file I am using for testing. Other large files work fine. I don't know exactly what's wrong with the culprit, but it seems as if the file size is much larger than the actual file contents. Weird.
 

Mel_Flynn

Well-Known Member

Reaction score: 81
Messages: 379

Sparse file maybe? Tho, EOF should be at what stat(2) says. I'd copy it to another dir. It's possible the directory entry is lying. If so, do fsck -y in single user.
 
Top