Reading files larger than 606666752 bytes

[noobster]

I'm trying to read a large file in C using open() and read(). I am doing this is a non-blocking way using a kqueue. Everything works fine until 606666752 bytes have been read from the file descriptor, but at that point kevent stops returning (i.e., no bytes are ready to be read). For smaller files, this is not an issue. Is there any setting I have to change to be able to read large files? Thanks in advance.

[Mel_Flynn]

Are you testing for EV_ERROR or similar?

FreeBSD off_t should handle up to OFF_MAX, which is nowhere near the value you're seeing. Would be easier to see your code...

[noobster]

If I run the code posted in the other thread (https://forums.freebsd.org/showpost.php?p=20582&postcount=3) on a large file, for some reason it hangs at 606666752 bytes. Any idea why?

[graudeejs]

check if there is limit set in /etc/login.conf

[noobster]

All settings seem to be set to unlimited. Also, I'm sure I didn't edit this file.

[Mel_Flynn]

Let us know if this is the same problem as EOF, you may want to warnx("%lli", ke.data) just before going to the next loop sequence.

[noobster]

Ah, it looks like that ke.data contains the value 606666752 in the first loop instead of the correct file size. Therefore, it only reads 606666752 bytes. I suspect this is because of the limited size of that struct field, but I have no idea how to get around this limitation. Anyone?

[Mel_Flynn]

When ke.data - BUFSIZ reaches <=0, issue EV_CLEAR on the next invocation? Not sure it'll work, but from coarse reading of the code, EV_CLEAR should clear the kernel copy of the event. Then you need to solve the problem to detect when really EOF.

[noobster]

Thanks for the suggestion, but it didn't work.

[Mel_Flynn]

Is this file INT_MAX+606666752 bytes? Cause I can't reproduce it with a 640M file.

[noobster]

The file I'm using is 4901634048 bytes, which is 4.6 gigabytes. So the file is much bigger than INT_MAX.

[noobster]

It looks like the problem is with the file I am using for testing. Other large files work fine. I don't know exactly what's wrong with the culprit, but it seems as if the file size is much larger than the actual file contents. Weird.

[Mel_Flynn]

Sparse file maybe? Tho, EOF should be at what stat(2) says. I'd copy it to another dir. It's possible the directory entry is lying. If so, do fsck -y in single user.