Guitarsmith Richard Stanley says: The nut really is critical, and its simple appearance belies its importance and the work necessary to make one properly. As far as a tone contributor, you only hear it on the open strings. The difference in string tone between open and fretted notes — with any nut — the open string tones are always brighter than the fretted notes. So the fingerboard nut is less important to tone than to how it affects playability, because the fingerboard nut:
1) Sets the spacing of the strings on the neck.
I space the strings evenly from edge to edge, which means there's the same amount of clearance between them all, but the inside four strings wind up being a bit off-center toward the treble side, because the bass strings are fatter. So rather than space on the string centers — which crowds the bass strings and gives you less room to get around, I do edge to edge spacing. How far they're set in from the edges of the fingerboard determines what your overall spacing can be. If they're too close to the fingerboard edges, you can inadvertently push them off the ends of the fret. If they're in too far, you waste space in the width of the neck that the player has to play around. So generally — unless there's something weird going on, I put the high E string in 3/32 of an inch from the end of the fret on the treble side, and 4/32 on the bass side — regardless of the guitar — electric or acoustic.
2) Sets the string elevation and playability in first position via the elevation adjustment — almost to the total exclusion of the adjustments at the bridge.
You can have a pretty high setup at the bridge end resulting in high string elevations in the upper register, but, if the nut is regulated low, first position can still play nicely. Conversely, you can have a setup that looks low, but first position could play terribly if the strings are too high coming off the nut. The range of dimensions for the string elevations there is typically between 0.012" and 0.020" between the under side of the string and the top of the crown of the first fret, requiring the use of feeler gauges to do a proper job of regulating this important setup feature.
The important thing to remember is that the fingerboard nut sets up playability in first position.