Scanline sync replaces the normal FPS limiter. If you set a non 0 value in scanline sync, the FPS limiter value is ignored.
For scanline sync to work correctly, the game needs to run vsync OFF. That means either just plain old vsync OFF or fast sync. (Fast sync will work because it runs games with vsync OFF.)
OK, so now as to how to use it, you need to enter the scanline RTSS should try to sync the FPS limiter against. Using "0" is special and means "disable scanline sync." We'll call it "s-sync" from now on because I don't want to have to type the word "scanline" over and over again
The most useful way to use s-sync is to use a negative value. RTSS interprets this as an offset relative to the last scanline in your current monitor resolution. The goal of s-sync is to give you a way of syncing the FPS limiter against a scanline that is not visible. At 1080p, you have 1080 visible scanlines, plus a couple dozen more that are invisible. If you sync against an invisible one, that means tearing will be invisible. If the total amount of scanlines is 1130 for example, if you enter "-50", then you are trying to sync against scanline 1080, which is the last visible one. This is not perfect however, and in reality RTSS will be only able to sync against a scanline greater than the one you specified. So you need to experiment. -20, -30, -40, and so on, until you see the tearline disappear off the bottom of the screen.
The flush option makes s-sync more accurate. Without flushing, syncing can be very erratic and jittery. The tearing will probably not be located around the target scanline, but will jump up and down seemingly at random. Flush will make it more accurate. "1" is only available in DX 10, DX 11 and OpenGL games. It's very fast but can be less accurate. "2" also works in DX 9, DX 12 and Vulkan games, but is slower and requires a stronger GPU to work without stutter. If you use flush=1 in a DX9/12 or Vulkan game, it's treated as flush=2.
Yes, you can do "1/2 s-sync", by clicking the "scanline sync" button until it says "scanline sync x/2". This will cap your FPS to half your refresh rate. The "x*2" setting will do the opposite, it will cap to double your refresh rate.
If your GPU is not fast enough, or the game is too heavy and you can't get the tearing to be hidden, you can instead use fast sync in the nvidia panel, and sync against scanline "1". This will have more input lag (because you're syncing against the first scanline instead of the last one), but this will allow fast sync to hide the tearline. If you try to sync against a higher scanline, input lag goes down, but the possibility of fast sync stutter will increase.