Go, chapter by chapter · Lesson 8 of 62 · about 50 min
Core packages
A guided tour of the standard library you will use every week: strings, io, files and folders, errors, containers, sorting and hashing.
On this page · 12 sections
strings#
The strings package works on strings without changing them (strings are immutable); every function returns a new value.
| Function | Example | Result |
|---|---|---|
Contains(s, sub) | Contains("gopher", "ph") | true |
Index(s, sub) | Index("gopher", "ph") | 2 (or -1) |
HasPrefix / HasSuffix | HasPrefix("golang", "go") | true |
Split(s, sep) | Split("a,b,c", ",") | [a b c] |
Fields(s) | Fields(" go is fun ") | [go is fun] (splits on any whitespace) |
Join(parts, sep) | Join([]string{"a","b"}, "-") | a-b |
Repeat(s, n) | Repeat("ab", 3) | ababab |
Replace(s, old, new, n) | Replace("aaa", "a", "b", 2) | bba (-1 means all) |
ToLower / ToUpper | ToUpper("go") | GO |
TrimSpace(s) | TrimSpace(" hi\n") | hi |
To build a string piece by piece, use strings.Builder. Adding with += in a loop copies the whole string each time (O(n²)); a Builder grows a buffer like append does (O(n)). Its sibling, strconv, converts between strings and numbers (Atoi, Itoa, ParseFloat, FormatInt).
io: Readers and Writers#
Two tiny interfaces carry most data in Go:
type Reader interface { Read(p []byte) (n int, err error) }
type Writer interface { Write(p []byte) (n int, err error) }Files, network connections, HTTP bodies, compressed streams, strings.NewReader and bytes.Buffer all implement one or both. So a function written against io.Reader works with all of them, and your tests can pass a string instead of a real file.
io.Copy(dst, src)streams everything from a Reader to a Writer.io.ReadAll(r)reads everything into a[]byte(fine for small inputs).bytes.Bufferis an in-memory Reader and Writer at once.bufioadds buffering:bufio.NewScanner(r)reads line by line;bufio.NewReaderandbufio.NewWriterbatch reads and writes.
Files and folders#
| Task | Use |
|---|---|
| Read a whole small file | data, err := os.ReadFile("a.txt") |
| Write a whole file | os.WriteFile("a.txt", data, 0o644) |
| Stream a large file | f, err := os.Open(...), defer f.Close(), then bufio.NewScanner(f) |
| Create or append | os.Create, or os.OpenFile with flags |
| List a folder | entries, err := os.ReadDir(".") |
| Walk a tree | filepath.WalkDir(root, fn) |
| Build paths portably | filepath.Join("data", "a.txt") |
0o644 is a Unix permission: owner can read and write, everyone else can read.
errors#
errors.New("out of stock") makes a simple error value. Declare it once at package level when callers need to check for it:
var ErrNoStock = errors.New("out of stock")You will learn wrapping with %w, errors.Is and errors.As in the "Go since the book" lesson.
Containers and sorting#
The container packages give ready-made structures:
| Package | Structure | Typical use |
|---|---|---|
container/list | doubly linked list | LRU caches |
container/heap | heap algorithms over your slice | priority queues, Dijkstra |
container/ring | circular list | round-robin |
For sorting:
slices.Sort(s)sorts any slice of ordered values (newest, fastest, generic).slices.SortFunc(s, cmp)sorts with your comparison function.sort.Slice(s, less)is the older reflection-based version you will see in existing code.sort.Sort(x)sorts any type implementingsort.Interface(Len,Less,Swap).
Hashes and cryptography#
A hash function turns any data into a fixed-size fingerprint.
hash/crc32andhash/fnvare fast, non-cryptographic: good for checksums and hash tables.crypto/sha256is cryptographic: use it when someone might try to forge data.- SHA-1 and MD5 are broken for security; avoid them for anything new.
- Never store passwords with a plain hash; use
golang.org/x/crypto/bcryptor argon2.
Fast input for algorithm practice#
Competitive programming judges feed large inputs through stdin. Unbuffered fmt.Scan can be the reason a correct solution times out. The standard template:
in := bufio.NewReader(os.Stdin)
out := bufio.NewWriter(os.Stdout)
defer out.Flush() // forgetting this prints nothing
var n int
fmt.Fscan(in, &n)More examples#
Files and folders, hands on
The program works inside a temporary directory so it never touches your real files. os.WriteFile and os.ReadFile cover most needs; os.Create plus a bufio.Writer is for writing piece by piece; filepath.WalkDir visits a whole tree.
package main
import (
"bufio"
"errors"
"fmt"
"io/fs"
"os"
"path/filepath"
"strings"
)
func main() {
dir, err := os.MkdirTemp("", "golesson-")
if err != nil {
panic(err)
}
defer os.RemoveAll(dir) // clean up everything at the end
// Whole-file write and read.
notes := filepath.Join(dir, "notes.txt")
if err := os.WriteFile(notes, []byte("line one\nline two\nline three\n"), 0o644); err != nil {
panic(err)
}
data, _ := os.ReadFile(notes)
fmt.Printf("read %d bytes\n", len(data))
// Line by line with a Scanner.
f, _ := os.Open(notes)
sc := bufio.NewScanner(f)
for n := 1; sc.Scan(); n++ {
fmt.Printf("%d: %s\n", n, sc.Text())
}
f.Close()
// Build a small tree, then walk it.
os.MkdirAll(filepath.Join(dir, "src", "util"), 0o755)
for _, name := range []string{"src/main.go", "src/util/strings.go", "src/util/README.md"} {
os.WriteFile(filepath.Join(dir, name), []byte("x"), 0o644)
}
filepath.WalkDir(dir, func(path string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
rel, _ := filepath.Rel(dir, path)
if strings.HasSuffix(path, ".go") {
fmt.Println("go file:", filepath.ToSlash(rel))
}
return nil
})
// A missing file is an error you can test for.
_, err = os.ReadFile(filepath.Join(dir, "missing.txt"))
fmt.Println("missing file is fs.ErrNotExist:", errors.Is(err, fs.ErrNotExist))
}read 29 bytes
1: line one
2: line two
3: line three
go file: src/main.go
go file: src/util/strings.go
missing file is fs.ErrNotExist: trueerrors.Is(err, fs.ErrNotExist) works through wrapping, which is why it is preferred over comparing error strings.
Sorting the book's people
The book sorts a slice of Person by name and then by age using sort.Sort with a custom type. The modern way is slices.SortFunc with a comparison function; both are shown so you can read older code.
package main
import (
"cmp"
"fmt"
"slices"
"sort"
)
type Person struct {
Name string
Age int
}
// ByName implements sort.Interface, the pre-generics way.
type ByName []Person
func (p ByName) Len() int { return len(p) }
func (p ByName) Less(i, j int) bool { return p[i].Name < p[j].Name }
func (p ByName) Swap(i, j int) { p[i], p[j] = p[j], p[i] }
func main() {
kids := []Person{{"Jill", 9}, {"Jack", 10}, {"Ana", 9}}
sort.Sort(ByName(kids))
fmt.Println("by name (sort.Sort):", kids)
slices.SortStableFunc(kids, func(a, b Person) int { return cmp.Compare(a.Age, b.Age) })
fmt.Println("by age, stable: ", kids) // Ana before Jill: their name order is kept
i, found := slices.BinarySearchFunc(kids, 10, func(p Person, age int) int { return cmp.Compare(p.Age, age) })
fmt.Println("first age-10 kid:", kids[i].Name, found)
}by name (sort.Sort): [{Ana 9} {Jack 10} {Jill 9}]
by age, stable: [{Ana 9} {Jill 9} {Jack 10}]
first age-10 kid: Jack trueHashes: checksums and fingerprints
The book compares two files with crc32 and introduces cryptographic hashes with sha1. Today, use CRC32 (or xxhash) to detect accidental corruption quickly, and SHA-256 whenever someone could tamper with the data on purpose. SHA-1 is broken for security uses.
package main
import (
"crypto/sha256"
"encoding/hex"
"fmt"
"hash/crc32"
"io"
"strings"
)
func checksum(r io.Reader) uint32 {
h := crc32.NewIEEE()
io.Copy(h, r) // a hash is an io.Writer: stream data into it
return h.Sum32()
}
func main() {
a := checksum(strings.NewReader("hello world"))
b := checksum(strings.NewReader("hello world"))
c := checksum(strings.NewReader("hello World"))
fmt.Println("same content, same crc:", a == b, " one letter changed:", a == c)
sum := sha256.Sum256([]byte("password123"))
fmt.Println("sha256:", hex.EncodeToString(sum[:])[:16]+"…")
fmt.Println("digest size:", len(sum), "bytes")
}same content, same crc: true one letter changed: false
sha256: ef92b778bafe771e…
digest size: 32 bytesThe full program#
package main
import (
"bufio"
"bytes"
"container/list"
"crypto/sha256"
"errors"
"fmt"
"hash/crc32"
"io"
"io/fs"
"os"
"path/filepath"
"slices"
"strconv"
"strings"
)
var ErrNoStock = errors.New("out of stock")
// countWords works with any Reader: files, network or strings.
func countWords(r io.Reader) (map[string]int, error) {
counts := map[string]int{}
sc := bufio.NewScanner(r)
sc.Split(bufio.ScanWords)
for sc.Scan() {
counts[strings.ToLower(sc.Text())]++
}
return counts, sc.Err()
}
func main() {
// strings and strconv
fmt.Println(strings.Fields(" go is fun "), strings.Split("a,b,c", ","),
strings.Repeat("ab", 3), strings.Index("gopher", "ph"))
var sb strings.Builder
for i := range 3 {
sb.WriteString(strconv.Itoa(i))
}
fmt.Println(sb.String())
// io
var buf bytes.Buffer
io.Copy(&buf, strings.NewReader("Go go GO stop"))
counts, _ := countWords(&buf)
fmt.Println(counts["go"], counts["stop"])
// files and folders
dir, _ := os.MkdirTemp("", "demo")
defer os.RemoveAll(dir)
path := filepath.Join(dir, "notes.txt")
if err := os.WriteFile(path, []byte("line 1\nline 2\n"), 0o644); err != nil {
fmt.Println(err)
return
}
f, err := os.Open(path)
if err != nil {
fmt.Println(err)
return
}
defer f.Close()
lines := bufio.NewScanner(f)
for lines.Scan() {
fmt.Println("read:", lines.Text())
}
filepath.WalkDir(dir, func(p string, d fs.DirEntry, err error) error {
if err == nild.IsDir() {
info, _ := d.Info()
fmt.Println("file:", d.Name(), info.Size(), "bytes")
}
return nil
})
_, err = os.ReadFile(filepath.Join(dir, "missing.txt"))
fmt.Println("missing file is an error:", err != nil)
// errors
fmt.Println(ErrNoStock)
// containers and sorting
l := list.New()
l.PushBack("b")
l.PushFront("a")
l.PushBack("c")
for e := l.Front(); e != nil; e = e.Next() {
fmt.Print(e.Value, " ")
}
fmt.Println()
nums := []int{5, 2, 9, 1}
slices.Sort(nums)
fmt.Println(nums)
// hashes
fmt.Println(crc32.ChecksumIEEE([]byte("gopher")))
fmt.Printf("%x\n", sha256.Sum256([]byte("gopher")))
}[go is fun] [a b c] ababab 2
012
3 1
read: line 1
read: line 2
file: notes.txt 14 bytes
missing file is an error: true
out of stock
a b c
[1 2 5 9]
875285291
9cc1ee455a3363ffc504f40006f70d0c8276648a5d3eb3f9524e94d1b7a83aefSummary#
strings,strconvandstrings.Buildercover text processing.io.Readerandio.Writerconnect files, networks and buffers; write functions against them.os.ReadFile,os.WriteFile,os.Openwithbufio.Scanner,os.ReadDirandfilepath.WalkDirhandle files and folders.container/list,container/heapandslices.Sortcover common data structures.- Use CRC32 for checksums, SHA-256 for integrity, and bcrypt for passwords.
Check your understanding
Answer before scrolling back up. If you miss one, reread the matching section, then try again.
Q1.Why do so many standard library functions accept an
io.Readerinstead of a file?Q2.What is the modern replacement for
ioutil.ReadFile?Q3.Why wrap
os.Stdininbufio.NewReaderwhen reading a million numbers?Q4.Which hash should you use to store something security-sensitive?
Practice in Go
Write a table test first, then the solution. On LeetCode, choose Go as the language. Open a hint only after 15 minutes stuck.
- Word frequency CLIbuild
Read a text file, count words case-insensitively with
strings.Fieldsand a map, sort by count, and print the top 10.Show hint
Write
count(r io.Reader) map[string]intso your test can passstrings.NewReader("a b a"). - Fast I/O benchmarkbuild
Generate a file with 1,000,000 numbers. Read it with
fmt.Scanand withbufio+fmt.Fscan, and time both.Show hint
Use
time.Since(start). Expect the buffered version to be many times faster. - Folder sizebuild
Use
filepath.WalkDirto total the size of every file under a folder and list the 5 largest.Show hint
d.Info()returnsfs.FileInfo, which hasSize(). - Ransom Note ↗easy
Show hint
Count letters of the magazine in a
[26]int, then subtract for the note. Show hint
XOR all bytes of both strings, or compare letter counts.
Finished this lesson?
Mark it done to track your progress. It is saved in this browser only.