06 - Input, Output & FS

1) File Descriptors

What & Why

On Unix, every open file, socket, or pipe is just a number called a file descriptor (FD). Think of it like a handle or ticket number given by the OS when you open something. Go’s *os.File is a wrapper around this FD, giving you safe methods. The first three FDs are always the same: 0 = stdin, 1 = stdout, 2 = stderr. Anything else (3, 4, 5…) are files or sockets you open.

fd-basic.go
package main

import (
	"fmt"
	"os"
)

func main() {
	f, err := os.Open("data.txt") // FD >= 3 typically
	if err != nil { panic(err) }
	defer f.Close()

	fmt.Println("OS FD:", f.Fd()) // e.g., 3
}
Note

Cross-platform

On Unix, a file descriptor is always a small integer given by the kernel (0,1,2,…). Every file, socket, or pipe is just another FD number. On Windows, the kernel doesn’t use FDs but HANDLEs (opaque pointers). Go fakes the FD API for portability: f.Fd() on Windows actually returns the underlying HANDLE value. Standard streams (stdin=0, stdout=1, stderr=2) are simulated for consistency. The key point: on Unix, FDs are real numbers; on Windows, they’re compatibility shims around native handles.
Note

Lifecycle

Always call Close(). If you forget, the file descriptor stays open until the garbage collector runs, which can leak resources. On a busy server, leaking FDs means 'too many open files' errors. So: open => use => close. Simple rule, critical in production.

2) File Offsets & Seeking (lseek)

Seek basics

  • (*os.File).Seek(off, whence) changes the 'cursor' position in a file. This offset decides where the next read/write happens.
  • Seek(off, io.SeekStart) => jump to off bytes from the beginning of the file.
  • Seek(off, io.SeekCurrent) => move relative to where you are now (negative values go backwards).
  • Seek(off, io.SeekEnd) => move relative to the end of the file. Example: -10 means 'go to 10 bytes before EOF'.
Note

Key points

Seek only works on files that have a definite length (like regular files). Pipes, sockets, and terminals don’t support it. Also: if a file is opened with O_APPEND, writes always go to the end, ignoring the current offset. But Seek still affects reads.
seek-example.go
package main

import (
	"fmt"
	"io"
	"os"
)

func main() {
	f, _ := os.Open("data.txt")
	defer f.Close()

	// Move to 10 bytes before EOF
	pos, _ := f.Seek(-10, io.SeekEnd)
	fmt.Println("new position:", pos)

	buf := make([]byte, 10)
	n, _ := f.Read(buf)
	fmt.Println("last bytes:", string(buf[:n]))
}

3) Open Flags

Core flag

  • O_RDONLY, O_WRONLY, O_RDWR => pick one access mode: read-only, write-only, or both.
  • O_CREATE => create the file if it does not exist.
  • O_EXCL => with O_CREATE, fail if the file already exists. This gives you an atomic 'create new'.
  • O_TRUNC => truncate the file to zero length if opened with write access (O_WRONLY or O_RDWR).
  • O_APPEND => every write is forced to the end of the file, no matter what the current offset is.
openfile-examples.go
package main

import (
	"log"
	"os"
)

func main() {
	// RW, create if missing
	f1, err := os.OpenFile("data1.txt", os.O_RDWR|os.O_CREATE, 0644)
	if err != nil {
		log.Fatal(err)
	}
	defer f1.Close()

	// Exclusive create (fail if exists)
	f2, err := os.OpenFile("data2.txt", os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0644)
	if err != nil {
		log.Println("exclusive create failed:", err)
	}
	if f2 != nil {
		f2.Close()
	}

	// Create or truncate
	f3, _ := os.OpenFile("data3.txt", os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0666)
	f3.Close()
}

4) Permissions, Special Bits & umask

Octal perms & umask

Permissions are expressed in octal (base 8). Split into three groups: user (owner), group, other. Each group has 3 bits: r=4, w=2, x=1. Add them up for a digit. Example: 6 = rw-, 7 = rwx, 5 = r-x.

  • 0644 => user=rw-, group=r--, other=r--.
  • 0755 => user=rwx, group=r-x, other=r-x.
  • Final mode = requested &^ umask. The umask is a process-wide 'bit filter' the OS applies, usually 022, which removes group/other write.
  • Leading 0 means octal. In Go, 0755 and 0o755 are the same literal.
perms_show.go
package main

import (
	"fmt"
	"os"
)

func main() {
	// Request 0644; actual will be 0644 &^ umask
	// 0644 &^ 0022 = 0644
	f, err := os.OpenFile("file.txt", os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0644)
	if err != nil {
		panic(err)
	}
	f.Close()

	info, _ := os.Stat("file.txt")
	// Print just the permission bits in octal (e.g., 0644)
	fmt.Printf("actual perms: %04o\n", info.Mode().Perm())
}
Note

Think of umask as subtraction

By default, Unix requests 0666 for new files (rw-rw-rw-) and 0777 for directories (rwxrwxrwx). The umask then removes bits. With umask 022: files become 0644, dirs become 0755. Technically it's a bitwise clear (&^), but subtraction is the easy mental model.
Tip

Remember

Your perm argument in Go is a maximum. The OS’s umask decides the final result. Go does not override it. Don’t be surprised if you ask for 0777 but get 0755.

Files vs Directories - what `x` really means

  • On files: x means you can execute it as a program or script.
  • On directories: x means you can cd into the directory and access its children (with proper read).
  • r on a dir lets you list names. x lets you enter. w lets you create/delete/rename inside. Usually you need both w and x together to modify entries.
dir_perms.go
package main

import (
	"fmt"
	"os"
)

func main() {
	// 0755: rwx for user, r-x for group/other => can traverse
	if err := os.Mkdir("pub", 0755); err != nil && !os.IsExist(err) {
		panic(err)
	}

	// 0700: only owner can enter
	if err := os.Mkdir("private", 0700); err != nil && !os.IsExist(err) {
		panic(err)
	}

	fi, _ := os.Stat("private")
	fmt.Printf("private perms: %04o\n", fi.Mode().Perm())
}
// private perms: 0700

Changing permissions at runtime (`Chmod`)

os.Chmod(path, mode) sets permissions. Combine with Stat().Mode().Perm() to read what you currently have. Typical use: make a script executable after writing it.

chmod_exec.go
package main

import (
	"fmt"
	"os"
)

func main() {
	os.WriteFile("tool.sh", []byte("#!/bin/sh\necho hi\n"), 0644)

	// Add user-exec bit: 0644 => 0744
	if err := os.Chmod("tool.sh", 0744); err != nil {
		panic(err)
	}

	st, _ := os.Stat("tool.sh")
	fmt.Printf("tool.sh perms: %04o\n", st.Mode().Perm())
}
// tool.sh perms: 0744

Reading modes & types (stat)

Go’s FileMode combines two things: the permission bits and the type bits. Use info.Mode().Perm() to isolate the 9 permission bits. Use io/fs masks like ModeSymlink, ModeSocket, etc. to test for type.

modes_types.go
package main

import (
	"fmt"
	"io/fs"
	"os"
)

func main() {
	info, err := os.Lstat("file.txt")
	if err != nil {
		panic(err)
	}

	mode := info.Mode()
	fmt.Printf("perms: %04o
	", mode.Perm())

	switch {
		case mode.IsDir():
		fmt.Println("type: directory")
		case mode.IsRegular():
		fmt.Println("type: regular file")
	}

	if mode&fs.ModeSymlink != 0 {
		fmt.Println("type: symlink")
	}
	if mode&fs.ModeSocket != 0 {
		fmt.Println("type: socket")
	}
	if mode&fs.ModeNamedPipe != 0 {
		fmt.Println("type: FIFO")
	}
	if mode&fs.ModeDevice != 0 {
		fmt.Println("type: device")
	}

	if mode&fs.ModeSetuid != 0 {
		fmt.Println("setuid set")
	}
	if mode&fs.ModeSetgid != 0 {
		fmt.Println("setgid set")
	}
	if mode&fs.ModeSticky != 0 {
		fmt.Println("sticky set")
	}
}

5) Managing Files

WriteFile & ReadFile

read-write-file.go
package main

import (
	"fmt"
	"os"
)

func main() {
	// Similar to: f, err := os.OpenFile(name, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, perm)
	os.WriteFile("out.txt", []byte("hello"), 0644)

	b, err := os.ReadFile("out.txt")
	if err != nil {
		panic(err)
	}
	fmt.Println(string(b))
}

Streaming

Note

What is streaming & why use it?

Streaming means handling data in small chunks instead of loading everything into memory. It's essential for huge files, logs, or sockets (think 10GB log).
streaming.go

package main

import (
	"bufio"
	"fmt"
	"io"
	"os"
)

func lineByLine(path string) error {
	f, err := os.Open(path)
	if err != nil {
		return err
	}
	defer f.Close()

	sc := bufio.NewScanner(f)
	// enlarge token limit if needed
	buf := make([]byte, 0, 1<<20)
	sc.Buffer(buf, 1<<20)

	for sc.Scan() {
		fmt.Println(sc.Text())
	}
	return sc.Err()
}

func copyFile(dst, src string) error {
	in, err := os.Open(src)
	if err != nil {
		return err
	}
	defer in.Close()
	out, err := os.Create(dst)
	if err != nil {
		return err
	}
	defer out.Close()

	buf := make([]byte, 1<<20) // 1MiB
	_, err = io.CopyBuffer(out, in, buf)
	return err
}

func main() {
	_ = lineByLine("big.log")
	_ = copyFile("b.bin", "a.bin")
}
Note

Why this is streaming

Both lineByLine and copyFile only keep a small buffer of data in memory at once (a line or a 1 MiB chunk). This lets you process arbitrarily large files or even infinite streams without ever running out of RAM. In contrast, os.ReadFile slurps the whole file into memory.

6) Console Input & Output

Reading a number, string (token), and float with fmt.Fscan

fmt.Fscan reads whitespace-separated tokens. Great for quick numeric input (like competitive programming).

scan-basic.go
package main

import (
	"fmt"
	"os"
)

func main() {
	var n int
	var name string
	var price float64

	fmt.Fprint(os.Stdout, "Enter: <int> <word> <float> => ")
	if _, err := fmt.Fscan(os.Stdin, &n, &name, &price); err != nil {
		fmt.Fprintln(os.Stderr, "scan error:", err)
		return
	}
	fmt.Printf("n=%d name=%q price=%.2f\n", n, name, price)
}

Reading a full line (with spaces)

To capture a whole line of input (with spaces), you usually use a bufio.Scanner. It’s the idiomatic and simplest choice: it splits by newlines and removes the \n or \r\n. If you need more control (like custom delimiters), you can use bufio.Reader.ReadString, but that returns the newline too, so you must trim it.

scan-line-scanner.go
package main

import (
	"bufio"
	"fmt"
	"os"
)

func main() {
	scanner := bufio.NewScanner(os.Stdin)
	fmt.Print("Say something: ")
	scanner.Scan()
	line := scanner.Text() // newline is already removed
	fmt.Println("You said (scanner):", line)
}
scan-line-reader.go
package main

import (
	"bufio"
	"fmt"
	"os"
	"strings"
)

func main() {
	in := bufio.NewReader(os.Stdin)
	fmt.Print("Say something: ")
	line, err := in.ReadString('\n') // includes the newline
	if err != nil {
		fmt.Fprintln(os.Stderr, "read error:", err)
		return
	}
	line = strings.TrimRight(line, "\r\n") // trim newline for cross-platform
	fmt.Println("You said (reader):", line)
}

Reading a single character (Unicode-safe)

Use ReadRune to correctly read a Unicode code point (rune). Don’t use ReadByte if you care about non-ASCII input.

scan-rune.go
package main

import (
	"bufio"
	"fmt"
	"os"
)

func main() {
	r := bufio.NewReader(os.Stdin)
	fmt.Print("Press a key and Enter: ")
	ch, size, err := r.ReadRune()
	if err != nil {
		fmt.Fprintln(os.Stderr, "read rune error:", err)
		return
	}
	fmt.Printf("rune=%q codepoint=U+%04X bytes=%d\n", ch, ch, size)
}

Writing to stdout & stderr

Send normal program output to stdout; send problems or logs to stderr. This separation lets users pipe your output without mixing errors.

stdout-stderr.go
package main

import (
	"fmt"
	"os"
)

func main() {
	fmt.Fprintln(os.Stdout, "normal output")
	fmt.Fprintln(os.Stderr, "error message")
}

7) Environment Variables

Reading & writing env

Environment variables are OS-level KEY=value pairs passed to your process (e.g., PATH, HOME, PORT). Use them for configuration so code and secrets aren’t hard-coded.

env.go
package main

import (
	"fmt"
	"os"
)

func main() {
	// Set and get
	_ = os.Setenv("APP_MODE", "dev")
	fmt.Println("APP_MODE:", os.Getenv("APP_MODE"))

	// Lookup with boolean
	if v, ok := os.LookupEnv("PORT"); ok {
		fmt.Println("PORT is", v)
	} else {
		fmt.Println("PORT not set")
	}

	// Enumerate all (KEY=VALUE)
	for _, e := range os.Environ() {
		fmt.Println(e)
	}
}
Tip
On Windows and Unix, env vars are inherited by child processes. Avoid storing secrets in code - prefer env vars or a secrets manager. In containers, inject them at runtime.