"NTFS really isn't that bad" - Robert Collins (LCA 2020)

Channel:linux.conf.au · Length:48:04 · Published:2020-01-16 · Views:160893

Watch on YouTube

Summary

Robert Collins explains why rustup installs were painfully slow on Windows and how changes to rustup and Windows cut install time from 3m30s to 14s. The talk covers NTFS, small files and the cost of antivirus hooks on file close.

Why HN recommends it

The go-to reference for why file operations are slow on Windows and how antivirus filters, not NTFS itself, are the cause.

This is a great talk on the topic — HN, 2024

C, C++, Rust, Go & systems languages · Software performance & optimization · Operating systems & kernels

Subtitles (excerpt)

okay today I bring you Robert Collins with an T and NTFS isn't really that bad [Applause] thank you for coming to listen to my motel and I hope you enjoy it we're going to talk about a tenth highest performance improvement and rust up on windows well most thing on Windows and before I get into this who here doesn't know what rust up is okay I should probably put my hand down so at any point during this talk questions are fine I don't want to leave anyone behind if you don't understand the rest of the talk that's going to be pretty disappointing but if I ask you to wait to the end or come and talk to me later please respect that there's a lot of people in the room discussions definitely they can wait I do value the feedback but there's a you know hundreds of so people they'll be processing whatever discussion we have them rather than just you and me come and see me after the talk now there will all the hands that came up when I asked about rust up and excuse me for once even the beard is not whether it was last year when I presented rust up is the language specific installer for rust so if you're not getting rust from your Linux distribution you get rust from rostov so it knows how to install rust and the

On Hacker News

And as a result of this, if you want to optimize your program to be fast on Windows, you will make very different optimization decisions than on other platforms. For example, Windows runs virus scans on close(), which makes it very slow. This means that sometimes it makes sense to have one or more background threads exclusively dedicated to closing files. There's a good talk on this at [VIDEO] — miki123211, 2025-01
I want to take this opportunity to recommend the talk "NTFS isn't that bad" ( [VIDEO] ). NTFS prefers a different access pattern than most usual file systems. I remember that a part of the talk was about speed-ups on Linux as well. So even if it doesn't sway your opinion it should enhance your perspective on how file systems work. — ablob, 2024-07
That may be due to a combination of Malware detection + most unix programs not really written to take advantage of the features NTFS has to offer This is a great talk on the topic [VIDEO] — unchar1, 2024-07
NTFS can be especially awful if you're used to Linux behaviour, port code over to Windows and expect the same performance. Here's a talk on porting rustup to Windows: [VIDEO] To begin with, it takes rustup 3m30s to install on Windows. After rejigging the code again and again away from its naive implementation which works fine on Linux, to perform sympathetically towards NTFS, it takes the same ru… — amiga386, 2024-07
Add your npm directories to the virus file exclusion lists of Windows Defender. Note that some of the directories may not be obvious (those under AppData for example). The speed improvements are massive for tools like these that touch lots of small files. Also take a look at this talk on NTFS, file system speed, and how the presenter optimized the Rust installer: [VIDEO] . There is an awful lot t… — jmmv, 2022-09

Related videos