NTFS or FAT32?

[Background Noise]

Having only recently bought a new dual core acer laptop I was surprised to see the hard disc is FAT32 rather than NTFS. Why would this be and are there any advantages to using NTFS (which I did on the previous one)

[Saab Dastard]

Errrr...

None. Have a look at this (http://www.microsoft.com/windowsxp/using/games/expert/durham_fs.mspx).

As to why? Probably easier for the manufacturer to install.

SD

[IO540]

A couple of years ago I did a lot of benchmarking between FAT16, FAT32 and NTFS, using a special program which allowed the filesize to be adjusted, etc.

NTFS was *much* slower when accessing lots of small files, say 1k in size. So slow in fact that it was not possible to write them to a 32x CD writer, without creating an intermediate image. This was a top-spec 10,000rpm SCSI hard disk fed off a £300 Adaptec SCSI controller, using the fastest available SCSI interface.

I now install FAT32 every time.

The main advantages of NTFS over FAT32 are

File size not limited to 4GB

Much better security; can have much finer granularity on permissions (irrelevant to most private users)

But for speed, FAT32 is probably better. I recently set up a special PC (3GHz, 1GB RAM, 100GB 10,000rpm SATA HD, top-end video card) mainly to run Pinnacle 10 for transferring HD data (over 1394) from a Sony HD HC1 cam, and it works better when writing the stuff to a FAT32 partition than an NTFS one.

BTW I am not happy about that other thread getting locked, especially as nobody actually answered my question.

[fabrifx]

fat 32 wastes a lot of space on the hard disk just to memorize the File Allocation Table at the beginning of the hard disk after the MBR. it's a bit complicated to say how a fat works here but it isn't a very good file system..let's say, it's not very efficient as NTFS or as the linux file systems are.. if anybody studied operating systems or C language, they'll understand me..this is a fat: it's a table made of two columns, on the first there are all the blocks of the hard disk, on the second, there are pointers or blocks to which the first blocks are linked..let's say a file is allocated in 3 blocks on the hard disk, 10, 11, 12.. in the fat beside the 10 there will be 11, beside 11 a 12, beside 12 an EOF(end of file).. when u open that file, it'll start reading block 10, since it points to block 11, it'll then read block 11 and this points to 12 so it'll read 12 and then it'll see EOF so it stops. this table takes a lot of unnecessary space.
:)

[Mac the Knife]

NTFS is not much slower that FAT32 on drives with a built-in cache (almost all these days) and far more efficient. Furthermore it's a journalled file system (like ext3 or Reiser FS) so is much more robust in the face of damage. FAT32 has effectively no security, while NTFS has excellent, fine-grained security. On todays big drives FAT32 is an inefficient anachronism.

Thanks to MS, the structure of NTFS is is a closed proprietary secret, so other operating systems have trouble accessing NTFS drives. Linux and other OSes can read NTFS alright, but writing is more problematic. Various schemes exist to allow other OSes to read/write NTFS - some of them use a wrapper for the MS drivers themselves and are pretty reliable, but this is not sanctioned by MS, though if you have Windows installed on another partition it's probably legal.

The only time I'd use FAT32 nowadays would for for a local partition that I needed to access regularly from Linux and Windows. Even then, I'd probably use ext3 and use ext2fsd (an installable file system driver for Windows) to allow me to access the ext3 partition from Windows rather than vice-versa.

NTFS is an excellent filesystem, it's just a pity that MS are so bent on preventing interoperability with other systems that they won't even release a binary driver, let alone publish the specifications.

NB:
It's exactly the same problem with the MS SMB/CIFS network system - MS doesn't want other systems to be able to access Windows systems over a network. MS wants you to chuck your other system and buy Microsoft, so they keep the structure of SMB/CIFS a closely guarded secret.

Happily Andrew Tridgell analysed SMB/CIFS (an enormous task) and wrote the open SAMBA protocol which allows other systems to interface with MS's SMB/CIFS. This works perfectly and the SAMBA team probably now know more about SMB/CIFS than Microsoft!

PS: There is an alternative to SAMBA and SMB/CIFS. Linux/UNIX supports NFS (Network File System) natively and if you install Microsoft's Services for Unix, Windows can too. I found it much harder to set up than SAMBA and never really got it going satisfactorily (if I'd RTFM I'd probably have been more successful!).

[tallsandwich]

All jolly interesting to read, I am not sure we answered one of the questions posted...

Your laptop will almost certainly be I/O limited when it appears so slow down - that is you will be normally be waiting for disk reads/writes to occur (regardless of how much memory you have). Exactly how that I/O wait is triggered depends on what you are doing (which applications you run and what you do with them). That is almost impossible to predict on a laptop as the use would be so varied, and is impacted by the applications, their configuration, the size of RAM and size of paging files etc etc.

Why would you want the fine grained security of NTFS if you bought a home computer - had you purchased a laptop for business/corporate/office use they might have considered that you would connect it to a MS domain and hence want to make use of those security features when sharing data.

Most likely FAT32 was the easiest and most flexible filesystem image to dump on the disk for the vendor, and given that the sales channel you used was targetted at home use, as opposed to business/corporate network use, this is most likely the explanation as to why you found FAT32.

Unless you can describe an accurate set of use scenarios that represent how you are going to use your laptop (almost impossible to do) and then run parallel tests, one on NTFS and the other on FAT32, then you will have no idea which will actually perform best. But you asked why FAT32 (not just which was better) and the answer is almost certainly that it was easier for the vendor :}

I agree with the above that NTFS is always a better choice just for reliability. Can anyone remember the NT "convert" command to change the filesystem to NTFS? It's been 8 years since I used it....

[IO540]

All very good tech points for NTFS, but not really relevant to a single user scenario with unlimited physical access by that user, and (one assumes) zero physical access for everybody else.

1. Caching is of no help with lots of small files being read for the first time in ages (hence my CD writing example)

2. NTFS security is exactly zero unless you have rock solid physical (access) security. If somebody can stick a DOS6.2 boot disk (or a suitable boot CD) in, with NTFSDOS on it, you can forget all security... So, no relevance to personal users. Equally, if an attacker can execute code of his choice (the case with most of M$ "buffer overflow" back doors) security is back to zero.

3. Efficiency doesn't come into it, with today's massive hard drives.

4. I wouldn't agree that NTFS is better for reliability, in the data recovery sense. I think NTFS does support full journalling but doesn't actually implement it (it would be impractical). So the data is stored only once and if you lose that bit you have lost it - just like FAT16, FAT32, etc. The worse thing, with a crashed drive, is that with FAT you can retrieve anything undamaged on the HD easily whereas NTFS is a right jumble.

[Saab Dastard]

tallsandwich, I think I did answer the "why it was installed" question in my first response, although without explaing why it would be easier.

The link I posted also had a comparative performance test between FAT32 and NTFS - OK, sponsored By MS - that indicated that there was less than 2% performance difference.

I can think of several reasons why NTFS on a home PC is desirable, if not essential!

To my mind, the most compelling reason to use NTFS on a home PC is to prevent malware from accessing the system files when connected to the internet.

With NTFS and logged on as a USER, rather than Admin, it is extremely unlikely that any system files can be modified. Not totally waterproof (mixing metaphors, here I think), but every little helps.

There is simply no equivalent protection with FATxx.

The second important reason for a home PC to run NTFS is for multiple users in a family to have private areas (agreed, accessible to admin equivalent, and susceptible to physical attack with various "hacking" tools).

It is also very important to have children's access restricted to USERS so that they cannot (again, caveats apply) inadvertently break the OS.

There is nothing with FAT32 to prevent system files being deleted. Please correct me if I'm wrong - I have never run an NT system since 3.51 without NTFS, and the last time I used FAT32 was with Win98!

Fair enough, if you are the only person in the household, then only the first reason is important - and if you connect to the internet as administrator then not even that. But I believe that it is less secure.

SD

[tallsandwich]

Good points, Sad Bastard :ok:

[Mac the Knife]

Hre ye go tallsandwich

Convert
Converts FAT and FAT32 volumes to NTFS

Syntax:
convert /fs:ntfs [/cvtarea:FileName] [/nosecurity] [/x]

Parameters
volume Specifies the drive letter (followed by a colon), mount point, or volume name to convert to NTFS. /fs:ntfs Required. Converts the volume to NTFS. /v Specifies verbose mode, that is, all messages will be displayed during conversion. /cvtarea:FileName For advanced users only. Specifies that the Master File Table (MFT) and other NTFS metadata files are written to an existing, contiguous placeholder file. This file must be in the root directory of the file system to be converted. Use of the /CVTAREA parameter can result in a less fragmented file system after conversion. For best results, the size of this file should be 1 KB multiplied by the number of files and directories in the file system, however, the convert utility accepts files of any size.

Important
•You must create the placeholder file using the fsutil file createnew command prior to running convert. Convert does not create this file for you. Convert overwrites this file with NTFS metadata. After conversion, any unused space in this file is freed. For more information about the fsutil file command, see Related Topics.

/nosecurity Specifies that the converted files and directory security settings are accessible by everyone. /x Dismounts the volume, if necessary, before it is converted. Any open handles to the volume will no longer be valid.

Remarks
•You must specify that the drive should be converted when the computer is restarted. Otherwise, you cannot convert the current drive.
•If convert cannot lock the drive (for example, the system volume or the current drive), it offers to convert the drive the next time the computer restarts.
•The location of the MFT is different on volumes that have been converted from previous version of NTFS, so volume performance might not be as good on volumes converted from Windows NT.
•Volumes converted from FAT to NTFS lack some performance benefits compared to volumes initially formatted with NTFS. On converted volumes, the MFT might become fragmented. In addition, on converted boot volumes, NTFS permissions are not applied after the volume is converted.

PS: NTFS is always journalled. You can remove some of its overhead by disabling its last file access update (which it does even when you so much as look at the contents of a directory). Go into Regedit and go to HKLM/System/CurrentControlSet/Control/FileSystem. Create a REG_DWORD called NtfsDisableLastAccessUpdate and set it to 1.

But I can't see any real point in doing this!

[tallsandwich]

NTFS is always journalled

But it seems IO540 thinks that althought it is journalled it is not 'implemented'. I always thought that the recovery features of NTFS were better (i.e. it could recover it's state more quickly and more accurately than FAT when a file system integrity check was required).

Journalling is just one of many ways to help acheive this recvoery process, right? There are many non journalled filesystems that have reasonably good recvoery mechanisms (for example good old UFS).

IO540, do you have any more info? I've seen many NTFS volumes recover from an inconsistient state (and recover wihout data loss) but I've never (to my knowledge) seen a FAT32 filesystem go successfully through this process.

If FAT32 is really just as good at recovering from inconsistent state as NTFS then I would certainly like to know. At the end of the day, on a single spindle system like most laptops, data reliability is almost certainly higher on the prioirty list for most people than marginal performance imporvements. Just to avoid confusion I am not talking about taking that failed volume and putting it in another device to facilitate recovery, I am talking about the self recovery of that single spindle machine without using specialist tools - that is the scenario that matters for 99% of laptop users.

Cheers.

[Mac the Knife]

NTFS is proprietary, so info about that goes on inside can only be deduced from "black box" techniques.

From - http://www.usenix.org/events/usenix05/tech/general/full_papers/prabhakaran/prabhakaran_html/main.html

"Although the source code or documentation of NTFS is not publicly available, tools for finding the NTFS file layout exist [28].

We ran the Windows XP operating system on top of VMware on a Linux machine. The pseudo device driver was exported as a SCSI disk to the Windows and a NTFS file system was constructed on top of the pseudo device. We ran simple workloads on NTFS and observed traffic within the SBA driver for our analysis.

Every object in NTFS is a file. Even metadata is stored in terms of files. The journal itself is a file and is located almost at the center of the file system. We used the ntfsprogs tools to discover journal file boundaries. Using the journal boundaries we were able to distinguish journal traffic from fixed-location traffic.

From our analysis, we found that NTFS does not do data journaling. This can be easily verified by the amount of data traffic observed by the SBA driver. We also found that NTFS, similar to JFS, does not do block-level journaling. It journals metadata in terms of records. We verified that whole blocks are not journaled in NTFS by matching the contents of the fixed-location traffic to the contents of the journal traffic.

We also inferred that NTFS performs ordered journaling. On data writes, NTFS waits until the data block writes to the fixed-location complete before writing the metadata blocks to the journal. We confirmed this ordering by using the SBA driver to delay the data block writes upto 10 seconds and found that the following metadata writes to the journal are delayed by the corresponding amount."

So NTFS is not a fully journalled file system (and there are different iinterpretations and implementations of "full") - so what? FAT32 has essentially no protection against file corruption and only a second FAT to "protect" against volume corruption (which is doesn't). FAT32 cannot recover from an inconsistent state - chkdsk which just discards anything that doesn't fit the primary FAT (which may itself be corrupt). NTFS can and does recover from inconsistent states, and remarkably well at that.

Roughly, the different kinds of journaling are:

1) data=writeback -- does no journalling of data; metadata only. fastest. data corruption possible in system crash.

2) data=ordered -- strictly speaking only journals metadata but uses transactions to protect data as well. slightly slower than writeback mode. Because of the way the blocks to be written to disk are ordered in this mode, it offers all the protection of full data journalling mode when files are being appended to; corruption is possible when files are being overwritten.

3) data=journal -- full journalling of data and metadata. Should be slowest because data is written twice, once to journal & once to final location. However some tests have shown that its performance surpasses the other two modes in busy interactive environments where disks are being read from & written to at same time. Also requires the largest disk footprint for journal.

So NTFS uses something on the data=ordered model - and so?

"I wouldn't agree that NTFS is better for reliability, in the data recovery sense."

Oh please.....

Finally, "..NTFS security is exactly zero unless you have rock solid physical (access) security..."

With physical access to ANY machine breaking in is trivial. Once in, it may even be possible to recover encryption keys and read 128-bit encrypted files (brute force won't do it). Without physical access to the machine(and assuming not-rooted), correctly set up NTFS permissions make seeing things you ain't meant to exceedingly difficult.

Don't blame NTFS (which is a nicely balanced filesystem) when the poor security model of the overlying OS is the real problem.

I'm not actually sure what you're complaining about to be honest.

[Saab Dastard]

Further NTFS advantages - even for the single user scenario - are file compression, volume mount points, directory junctions, hard links and symbolic links. There is also support for volume shadowing and sparse files.

There's more, but it begins to become somewhat esoteric!

SD

[tallsandwich]

So it would appear that unless Background Noise's laptop was being used for very specific high I/O requirements (where FAT32 has been shown by IO540 to outperform NTFS) then the filesystem on said laptop would be better being converted to NTFS. What do you think, Background Noise? Ready to convert? :eek: :}

[Background Noise]

Well I'm not sure about converting - I need to re-read the stuff above and/or research. I guess I was just surprised having thought that NTFS was the newer, and therefore better(:rolleyes:) system. I used it previously on my home laptop which I formatted and loaded from scratch. This one came preinstalled and I only discovered the format by right clicking the pre-partitioned drives. I might wipe he whole thing sometime though as I get hacked off with most of the supplied stuff - when I can work out how to keep/re-install the utilities I want from acer.

[Loose rivets]

Most of the discussion seems to have revolved around well managed discs. What of the long-term neglected disc, and the issue of fragmentation?

[Mac the Knife]

I guess I was just surprised having thought that NTFS was the newer, and therefore better(:rolleyes:) system.

Newer isn't always better, but I sort of thought we'd established in this discussion that NTFS was, for all except quite special situations, a great deal better than the old FAT32 filesystem.

I think IO540 is completely wrong.

Whatever tiny speed advantages that might possibly be gained under special situations from using a simple old filesystem are completely negated by the many advantages and much increased robustness of a journalled filesystem like NTFS. Why do you think people use journalled systems? To deliberately slow their machines down? Jeez!

As for the idea of using what is effectively an obsolete filesystem on a new dual-core notebook...words fail me!

:ugh:

PS: My wife's Asus notebook also came with the primary partition formatted as FAT32. I've no idea why they do this. First thing I did was convert is to NTFS.

PS: Loose rivets - NTFS also fragments, but the fragmentation has less effect on speed, so defragging all the time isn't so important.

[Background Noise]

I hadn't realised that the file system could be changed without reformatting.

[goates]

PS: My wife's Asus notebook also came with the primary partition formatted as FAT32. I've no idea why they do this. First thing I did was convert is to NTFS.

The manufacturers may be doing this to prevent people from locking themselves out of their computer or running into permission issues. If you happen to set the permissions wrong, or forget a password, you're in big trouble under NTFS. With FAT32 you (and any viruses or malware apps running) can get at every file without much trouble.

I know personally I've had cases where after upgrading my WinXP machine and reinstalling the system, I no longer had permissions to open some files. It isn't hard to fix if you know what you're doing, but I don't think the average person will find it easy.

[Saab Dastard]

The manufacturers may be doing this to prevent people from locking themselves out of their computer or running into permission issues.

Hmmm... Shouldn't the manufacturers be more concerned with securing PCs against attack from the internet?

It's a bit like saying car manfrs don't put locks on their cars against the possibility that someone might lose their keys.

SD

[goates]

Hmmm... Shouldn't the manufacturers be more concerned with securing PCs against attack from the internet?

It's a bit like saying car manfrs don't put locks on their cars against the possibility that someone might lose their keys.

SD

If you lose your keys, you can get your local auto club to open the door and then get new keys and/or locks. With the passwords on NTFS, you can't get them back, at least not by any means the average person could do. It would be a support nightmare.

[Mac the Knife]

Funny old world...

On one thread I find myself supporting Microsoft's proprietary NTFS filesystem and on the other I find myself supporting Linux!

Both good OSes IMHO, but with different approaches and philosophies.

And prices of course.....

goates

"With the passwords on NTFS, you can't get them back, at least not by any means the average person could do."

It's quite easy, unless your "average" user is very dim (admittedly, lots are).

Getting into a physically accessible Windows box when you've forgotten either your password or the Administrators password is trivial.

Get a friend to download and burn one of the innumerable Windows password reset disks available on the WWW and Bob's your uncle.

http://home.eunet.no/~pnordahl/ntpasswd/ is one of many.

No friends? Then you might have a problem....

BTW, MS does encourage people to make an "official" Password Reset Disk for just this situation.

PS: Getting in is one thing and reading encrypted files is quite another.
If you've encrypted your files then you're out of luck. Without the correct password modern encryting file systems are effectively unbreakable. But most people who encrypt files are aware of this and stash copies away safely.

[Loose rivets]

I mentioned this thread to a friend, and he e-mailed me back with this overview. It seems to contain some extra angles on the issue.

I edit to say that when he says

'FAT was always very unreliable;' it is from his view point. I recal him saying 25 years ago, 'a system should not crash....EVER.' not judged by mortal standard then. ;-)


So, the issues of NTFS vs. FAT continue. FAT (File Allocation Table)
was "invented" for the very first floppy drives. In those days it used
only 12 bit cluster addresses and thus could address only 4096 clusters
- more than enough for the tiny floppy disks of the time. When the
first
PC hard drives appeared they extended FAT to 16 bit addressing, and
this
was known as FAT16. This allowed 65536 clusters to be handled. The
maximum sized disk was then 65536 times the cluster size: a 4K cluster
gave a 1/4 Gig max. disk, and a 32K cluster gave the magic 2 Gig. max.
Unfortunately, each cluster can contain only one file, or portion of a
file. You cannot put two files in the same cluster, so a file
containing
only one byte occupies a complete cluster - if its a 32K cluster then
32k-1 bytes were wasted.

With the advent of FAT32, the 2Gig barrier was finally broken and
sensible sized clusters could be used, but that's not the end of the
story...

FAT was always very unreliable; one attempt to combat this was to
maintain a duplicate FAT (the actual FAT is, in essence, the heart of
the system - when it gets corrupted you can lose bits of files). When
the FATs got out of sync it was apparent that something had gone wrong
and you could run CHKDSK to fix it. This is the reason that you were
forced to run it when the system wasn't properly closed down. Despite
the duplicate FAT the whole system was never very reliable and files
were often found to be corrupted.

Enter totally different file systems. The best one of all was the one
that IBM developed for OS/2 - the one called HPFS (High Performance
File
System) - but this lost favour with the demise of OS/2. When MS's
version of OS/2 came out it had a brand new file system - NTFS (New
Technology File System). This was based on the HPFS system but wasn't
quite as good. There have been 5 versions of NTFS - 1 to 3 were used
for NT 1 to NT 3. version 4 was for Windows 2000, and version 5 for XP.
All the NTFSs broke the 2Gig barrier, but the earlier versions were not
particularly good performers. The later versions improved this and were
not very different from FAT32, but they were much more reliable.

In short, NTFS is FAR better than FAT. Its as fast and much more
reliable. There really is no contest. FAT was never designed for large,
permanently mounted hard disks. NTFS was. Period. The big problem is
that W98 and earlier cannot read it at all - NTFS partitions are simply
invisible to systems earlier than W98.

[Saab Dastard]

The big problem is
that W98 and earlier cannot read it at all

It is true that there is no native MS support for NTFS in the Win9x family: however, there are 3rd party add-ins that allow NTFS partitions to be read by Win9x OS's.

With the advent of FAT32, the 2Gig barrier was finally broken and
sensible sized clusters could be used

True - but only up to 32GB - after that, the block size is 32K

< = 512MB --512 Bytes
< = 8GB ---- 4 Kilobytes
< = 16GB --- 8 Kilobytes
< = 32GB --- 16 Kilobytes
> = 32GB --- 32 Kilobytes

Compare this to NTFS - block size is a constant 4KB by default (can be increased from 0.5 KB to 64 kB depending on the application).

Note that If you setup Windows on FAT and then convert the volume from FAT to NTFS, it usually causes Master File Table (MFT) fragmentation, so in general, converted partitions are somewhat slower than ones originally created as NTFS. The only solution to this is to backup everything, re-format and restore.

SD