There are several ways to read a text file block by block: 1) Use the HeaderLines option But on a text file it is more dangerous and confusing than anything ( unless you are absolutely sure that each line is the same size and each element on the line is at the same exact place/column, but that doesn't happen often). It is very useful for binary files when you just want to skip some records of known length. This doesn't seem to happen when using textscan in the "appropriate" way (either with or without ftell and fseek): in both cases it tries to continue from its last position, but to no avail because of some reason I don't understand yet.įseek a pointer in a file is only good when you know precisely where (or by how many bytes) you want to move the cursor. I'm not entirely sure I understand what HeaderLines does in this context, but it seems to make textscan completely ignore everything that comes before the specified line. This seems to read in the data without producing errors it is, however, incredibly slow. Then I tried using the HeaderLines solution (also suggested below), like this: i = 0 ĭata = textscan(fid, FormatString, nLines, 'Delimiter',' ', 'HeaderLines', i*nLines) This gave exactly the same error as I got before: MATLAB was unable to read in more than approximately 43% of the file.
MATLAB TEXTSCAN CODE
The relevant part of my code used to look like this: while ~feof(fid)ĭata = textscan(fid, FormatString, nLines, 'Delimiter', ' ') %// Read nLinesįirst I tried fixing it using ftell and fseek as suggested by Hoki below. Any suggestions would be greatly appreciated! The only possible explanation that I can think of is that something goes wrong on a much deeper level that I have little understanding of. I looked at the file using a text editor for large files, and this shows that the entire file looks fine, and that there should be no reason for textscan to be confused. moving it to about 50% of the file) everything breaks down and textscan only returns an empty 1x7 cell. (I'm aware this doesn't necessarily mean the indicator is at the beginning of a line, but if I run textscan twice I get a satisfactory answer.) Now, if I substitute tes/10 by tes/2 (i.e. I'm assuming that the way I'm using fseek here moves the position indicator to about 10% of my file. To debug, I attempted to go to several positions in the file using the fseek function, for example like this: fileInfo = dir(fileName) įseek(fid, floor(tes/10), 'bof') This only allows me to scan about 43% of the file, after which textscan starts returning empty cells (despite there still being data left to scan in the file). According to the documentation this should allow for processing "arbitrarily large delimited text file"). loading and analyzing a smaller block of the file at a time. The file in question is about 12 GB in size and contains about 250 million lines with seven (floating) numbers in each (delimited by a whitespace) because this obviously would not fit into the RAM of my desktop, I'm using the approach suggested in the MATLAB documentation (i.e.
I'm trying to analyze a very large file using textscan in MATLAB.
0 kommentar(er)
