csvutil

Package csvutil provides fast, idiomatic, and dependency free mapping between CSV and Go (golang) values.

This package is not a CSV parser, it is based on the Reader and Writer interfaces which are implemented by eg. std Go (golang) csv package. This gives a possibility of choosing any other CSV writer or reader which may be more performant.

Installation

go get github.com/jszwec/csvutil

Requirements

Go1.18+

Example <a name="examples"></a>

Unmarshal <a name="examples_unmarshal"></a>

Nice and easy Unmarshal is using the Go std csv.Reader with its default options. Use Decoder for streaming and more advanced use cases.

	var csvInput = []byte(`
name,age,CreatedAt
jacek,26,2012-04-01T15:00:00Z
john,,0001-01-01T00:00:00Z`,
	)

	type User struct {
		Name      string `csv:"name"`
		Age       int    `csv:"age,omitempty"`
		CreatedAt time.Time
	}

	var users []User
	if err := csvutil.Unmarshal(csvInput, &users); err != nil {
		fmt.Println("error:", err)
	}

	for _, u := range users {
		fmt.Printf("%+v\n", u)
	}

	// Output:
	// {Name:jacek Age:26 CreatedAt:2012-04-01 15:00:00 +0000 UTC}
	// {Name:john Age:0 CreatedAt:0001-01-01 00:00:00 +0000 UTC}

Marshal <a name="examples_marshal"></a>

Marshal is using the Go std csv.Writer with its default options. Use Encoder for streaming or to use a different Writer.

	type Address struct {
		City    string
		Country string
	}

	type User struct {
		Name string
		Address
		Age       int `csv:"age,omitempty"`
		CreatedAt time.Time
	}

	users := []User{
		{
			Name:      "John",
			Address:   Address{"Boston", "USA"},
			Age:       26,
			CreatedAt: time.Date(2010, 6, 2, 12, 0, 0, 0, time.UTC),
		},
		{
			Name:    "Alice",
			Address: Address{"SF", "USA"},
		},
	}

	b, err := csvutil.Marshal(users)
	if err != nil {
		fmt.Println("error:", err)
	}
	fmt.Println(string(b))

	// Output:
	// Name,City,Country,age,CreatedAt
	// John,Boston,USA,26,2010-06-02T12:00:00Z
	// Alice,SF,USA,,0001-01-01T00:00:00Z

Unmarshal and metadata <a name="examples_unmarshal_and_metadata"></a>

It may happen that your CSV input will not always have the same header. In addition to your base fields you may get extra metadata that you would still like to store. Decoder provides Unused method, which after each call to Decode can report which header indexes were not used during decoding. Based on that, it is possible to handle and store all these extra values.

	type User struct {
		Name      string            `csv:"name"`
		City      string            `csv:"city"`
		Age       int               `csv:"age"`
		OtherData map[string]string `csv:"-"`
	}

	csvReader := csv.NewReader(strings.NewReader(`
name,age,city,zip
alice,25,la,90005
bob,30,ny,10005`))

	dec, err := csvutil.NewDecoder(csvReader)
	if err != nil {
		log.Fatal(err)
	}

	header := dec.Header()
	var users []User
	for {
		u := User{OtherData: make(map[string]string)}

		if err := dec.Decode(&u); err == io.EOF {
			break
		} else if err != nil {
			log.Fatal(err)
		}

		for _, i := range dec.Unused() {
			u.OtherData[header[i]] = dec.Record()[i]
		}
		users = append(users, u)
	}

	fmt.Println(users)

	// Output:
	// [{alice la 25 map[zip:90005]} {bob ny 30 map[zip:10005]}]

But my CSV file has no header... <a name="examples_but_my_csv_has_no_header"></a>

Some CSV files have no header, but if you know how it should look like, it is possible to define a struct and generate it. All that is left to do, is to pass it to a decoder.

	type User struct {
		ID   int
		Name string
		Age  int `csv:",omitempty"`
		City string
	}

	csvReader := csv.NewReader(strings.NewReader(`
1,John,27,la
2,Bob,,ny`))

	// in real application this should be done once in init function.
	userHeader, err := csvutil.Header(User{}, "csv")
	if err != nil {
		log.Fatal(err)
	}

	dec, err := csvutil.NewDecoder(csvReader, userHeader...)
	if err != nil {
		log.Fatal(err)
	}

	var users []User
	for {
		var u User
		if err := dec.Decode(&u); err == io.EOF {
			break
		} else if err != nil {
			log.Fatal(err)
		}
		users = append(users, u)
	}

	fmt.Printf("%+v", users)

	// Output:
	// [{ID:1 Name:John Age:27 City:la} {ID:2 Name:Bob Age:0 City:ny}]

Decoder.Map - data normalization <a name="examples_decoder_map"></a>

The Decoder's Map function is a powerful tool that can help clean up or normalize the incoming data before the actual decoding takes place.

Lets say we want to decode some floats and the csv input contains some NaN values, but these values are represented by the 'n/a' string. An attempt to decode 'n/a' into float will end up with error, because strconv.ParseFloat expects 'NaN'. Knowing that, we can implement a Map function that will normalize our 'n/a' string and turn it to 'NaN' only for float types.

	dec, err := csvutil.NewDecoder(r)
	if err != nil {
		log.Fatal(err)
	}

	dec.Map = func(field, column string, v any) string {
		if _, ok := v.(float64); ok && field == "n/a" {
			return "NaN"
		}
		return field
	}

Now our float64 fields will be decoded properly into NaN. What about float32, float type aliases and other NaN formats? Look at the full example here.

Different separator/delimiter <a name="examples_different_separator"></a>

Some files may use different value separators, for example TSV files would use \t. The following examples show how to set up a Decoder and Encoder for such use case.

Decoder:

	csvReader := csv.NewReader(r)
	csvReader.Comma = '\t'

	dec, err := csvutil.NewDecoder(csvReader)
	if err != nil {
		log.Fatal(err)
	}

	var users []User
	for {
		var u User
		if err := dec.Decode(&u); err == io.EOF {
			break
		} else if err != nil {
			log.Fatal(err)
		}
		users = append(users, u)
	}

Encoder:

	var buf bytes.Buffer

	w := csv.NewWriter(&buf)
        w.Comma = '\t'
	enc := csvutil.NewEncoder(w)

	for _, u := range users {
		if err := enc.Encode(u); err != nil {
			log.Fatal(err)
		}
        }

	w.Flush()
	if err := w.Error(); err != nil {
		log.Fatal(err)
	}

Custom Types and Overrides <a name="examples_custom_types"></a>

There are multiple ways to customize or override your type's behavior.

a type implements csvutil.Marshaler and/or csvutil.Unmarshaler

type Foo int64

func (f Foo) MarshalCSV() ([]byte, error) {
	return strconv.AppendInt(nil, int64(f), 16), nil
}

func (f *Foo) UnmarshalCSV(data []byte) error {
	i, err := strconv.ParseInt(string(data), 16, 64)
	if err != nil {
		return err
	}
	*f = Foo(i)
	return nil
}

a type implements encoding.TextUnmarshaler and/or encoding.TextMarshaler

type Foo int64

func (f Foo) MarshalText() ([]byte, error) {
	return strconv.AppendInt(nil, int64(f), 16), nil
}

func (f *Foo) UnmarshalText(data []byte) error {
	i, err := strconv.ParseInt(string(data), 16, 64)
	if err != nil {
		return err
	}
	*f = Foo(i)
	return nil
}

a type is registered using Encoder.WithMarshalers and/or Decoder.WithUnmarshalers

type Foo int64

enc.WithMarshalers(
	csvutil.MarshalFunc(func(f Foo) ([]byte, error) {
		return strconv.AppendInt(nil, int64(f), 16), nil
	}),
)

dec.WithUnmarshalers(
	csvutil.UnmarshalFunc(func(data []byte, f *Foo) error {
		v, err := strconv.ParseInt(string(data), 16, 64)
		if err != nil {
			return err
		}
		*f = Foo(v)
		return nil
	}),
)

a type implements an interface that was registered using Encoder.WithMarshalers and/or

Csvutil

Install / Use

README