This is generally great advice, and aside from the package manager recommendations is still relevant today. I do take issue with a few things though.
1. Don't put your internal libraries in /pkg. /pkg has special GOPATH meaning as "compilation cache". Its not an actual name conflict, but why bother risking it.
2. This is just wrong:
> fmt.Printf is self-contained and doesn’t affect or depend on global state; in functional terms, it has something like referential transparency. So it is not a dependency. Obviously, f.Bar is a dependency. And, interestingly, log.Printf acts on a package-global logger object, it’s just obscured behind the free function Printf. So it, too, is a dependency.
stdout (and the buffer, and mutex on it) are exactly the same as the global log object. In fact, `log.Printf` is more or less just an alias to `fmt.Printf`[0]
3. I wish it had mentioned the functional options pattern in the part about constructors[1]
Functional options are controversial in the Go community. I don't like them personally. The idea of having a bunch of functions that exist only to mutate internal state on init is... an odd choice. Google code is riddled with this sort of style.
It's annoying to write client code for another reason: It's hard to discover. For the "term" example in the article, you can't sit in your editor/IDE and type "term." and get a list of setter suggestions, since everything in a single namespace. "term.Speed()" does not sound like an option to me. Function names are generally verbs, and having one called Speed() doesn't read well. And it causes issues if you want to have a type called Speed. Then it has to be term.WithSpeed() or term.SetSpeed() or something. Neither of which is really self-explanatory.
The fact that options are functions has other downsides. The moment you pack a option into a function, they've lost their ability to be introspected. You can't dump the options to a debug log before you invoke the constructor function. You can never ask an option function what it contains.
I prefer representing options using actual data:
type (
Option interface {
isOption()
}
Speed int
RawMode bool
SomeOtherSetting struct {
A, B int
}
)
Unfortunately, due to how Go interfaces work, you'll have to provide a dummy private method to tie the types together, but it's a small loss. (Go itself uses this pattern in a bunch of places, such as the compiler's AST.)
A big config struct is great, but it comes with a big downside: You have to rely on Go's zero value semantics to provide defaults. It's not always clear that 0 or an empty string means "no value".
It means that sometimes you'll have to use dummy values (-1 for "no value", for example) or pointers (nil means optional). It means that any boolean has to be phrased in such a way that false is the default (so if it feels natural to call the option UseKeychain and it should default to true, you have to invert it and call it DontUseKeychain).
Config structs also means it's easier for a caller to construct invalid configs:
type Config struct {
// Either stream or channel can be set, but not both
OutputStream io.Writer
OutputChan chan Event
}
// ...
pipelines.New(pipelines.Config{
OutputStream: myFile,
OutputChan: myChan,
})
The semantics cannot be guaranteed by the type system, so won't be caught until runtime. Neither will this, of course:
...but in this case we can quite sensibly mandate that the following option overrides the previous one, flipping the internal switch over to the setting you specify. With the Config struct, there's no way for pipelines.New() to know which one should have precedence.
(In this particular case, you can work around it by having a single Output field that uses an interface, but there are other situations where mutually incompatible fields can occur.)
Yep! It was probably overstepping on my part to say that the entire benefit depends on that argument. I should instead have said that the tradeoff does not seem good to me.
It depends on the purpose. Config structs are indeed simpler and make tons of sense for cases where you control all the related code. So yeah, for code where you can do a mostly-automated refactor if you need to change the API, I totally agree that Config structs make more sense and functional options are huge overkill.
For stuff that needs finer-grained control, or long-lived API stability, functional options give you a lot more flexibility.
> Functional options are controversial in the Go community. I don't like them personally. The idea of having a bunch of functions that exist only to mutate internal state on init is... an odd choice. Google code is riddled with this sort of style.
I come from a background in functional programming, and having used functional options in Go for a while now, I completely agree. I've never seen an implementation of them that felt natural or idiomatic to use. They're incredibly awkward, and I basically always prefer the alternative approaches to the functional options style.
> 1. Don't put your internal libraries in /pkg. /pkg has special GOPATH meaning as "compilation cache". Its not an actual name conflict, but why bother risking it.
It's a little too late for this because it's pretty much an unofficial standard for library code :-) Just look at Kubernetes, Docker or many other major Go-based applications/tools.
Last I checked, pkg is going away as a compilation cache. As of Go 1.10, the compilation is in fact now stored elsewhere: https://golang.org/doc/go1.10
I simply love the go toolchain. The defacto standards for code formatting, linting and vetting, and the speed of builds and tests save tons of time. Both clock time and overall developer productivity.
I have a GitHub CI service that verifies the standard go tools pass on every commit.
The bot itself is naturally written in Go and executed in a Lambda function.
Everything is just so fast! You get feedback in seconds. Which keeps you in the flow of coding.
The code from go kit and [oklog](https://github.com/oklog/oklog) are great examples of idiomatic Go.
Unfortunately the community at large doesn't really follow the "no init"/"no package global vars", which can sometimes lead to bad experiences importing opensource Go libs.
I feel like go-kit is quite antithetical to the Go mindset...it presents a lossy abstraction as a means of future-proofing against eventualities that will almost certainly never be encountered
to be honest it strikes me as the sort of library that excites intermediate developers who tend to over-architect
Your "almost certainly never" is another organization's "certainly inevitable" or "already happened". Go kit's scope and applicability is pretty clearly enumerated in the documentation. And there's nothing lossy about its abstractions.
> the sort of library that excites intermediate developers
I find this true about nearly all microservices in Go. Microservices are much more useful in something like Node that can only take advantage of 1 OS thread per instance. Without containerization and load balancing in Node you wouldn't be able to scale.
Go on the other hand can efficiently utilize a nearly unlimited amount of threads as necessary with its scheduler. You're much more likely to over-architect if you don't keep this capacity in mind.
Search for "update:" in the article. There are three and the most recent references a post from June of 2017. It appears the author has modified it for developments he considers important. It would be helpful if he indicated the revision date near the top of the article.
> Well, it is on its way out, have you missed the news regarding vgo?
Dep will have a clean migration path to vgo, and the latter isn't really production-ready yet.
For now, I'd recommend using dep for daily use, until you have a compelling reason to switch to vgo, at which point the migration will probably be automatic.
Yeah... In defense of the Go folks (1) they are working hard to make the transition as smooth as possible (2) dep was always just considered an experiment (though with aspirations by the author to have it become the official solution) (3) vgo almost certainly marks the end of the churn, and looks really nice.
> Go's dependency management story is starting to resemble JS's modules story
It's worse tbh. npm was actually always usable if you knew what you were doing. Complaints with npm were generally down to misunderstanding, people not being aware of features like 'npm shrinkwrap' (which has now been replaced with a lockfile-by-default strategy) and of course registry issues (allowing unpublishing) that weren't unique to npm, and were only an issue for people relying on public registries at deploy-time, a bad idea.
vgo is at least phrased as being a proposal, not necessarily the future of Go [1]:
This post sketches a proposal for doing exactly
that, along with a prototype demonstration ...
I intend this post to be the start of a productive
discussion about what works and what doesn't.
Based on that discussion, I will make adjustments
to both the proposal and the prototype, and then
I will submit an official Go proposal, for
integration into Go 1.11 as an opt-in feature.
Of course, this is Russ Cox, so chances are that his proposal will carry more weight with his fellow core Go team than that of Dep's authors.
Sam Boyer's follow-up is interesting reading [2]. I get the feeling that despite their ongoing discussions, the Dep team was/felt ambushed by this move.
We use Dep. It's good, much better than the buggy mess that is/was Glide.
My only criticism is that "dep ensure" will actually parse the code to discover dependencies through import statements, which is also what Glide does. To me, this is antithetical to the purpose of a Gopkg.toml/lock file. In other words, Dep's full list of dependencies isn't actually in the Gopkg.toml file; it's a sum of Gopkg.toml and your code. That is confusing.
My desired behaviour:
* "dep ensure" should always used the lock file, nothing else, to install;
* "dep ensure -update" should update the lock file to what is specified in Gopkg.toml (and only that);
* "dep ensure -add" (which I think should be "dep add") should b required to add new dependencies to the Gopkg.toml file.
Aside: I wish Go projects weren't stuck with BSD style flags (-update instead of --update). GNU style is more common and arguably more practical. I applaud whenever a project (e.g. Prometheus, recently) finally sees sense and goes over to GNU flags.
I'm fairly certain you can use `--flag` with the standard flag package from stdlib: https://godoc.org/flag (in the section on command line flag syntax).
Though supporting `-flag` does remove the very nice combining of short flags.
Interesting, I didn't know that. I never liked the flag package, and the downside you mention is a reason not to use it. Kingpin (used by the Prometheus projects) and go-flags are nicer. Lots of Google code uses spf13's pflag, which I'm not a fan of, but it does do GNU flags correctly.
The weird thing is if you -add a dependency before you import it, then you get a warning that the dependency isn't used in the codebase yet. Of course it isn't, I just added it! My IDE (IDEA) resolves the imports for me, so I can't add the import until after I add the dependency. Nice little catch-22.
Pretty good advice, except the preference for interfaces to support mocking. You should avoid mocking when possible in favor of real objects or fakes. Interfaces are still cool, though.
You don't test the http.Client as part of your business logic, even (especially) if your business logic makes HTTP requests. Instead, you trust that package http is already well-tested, mock out http.Client as an abstract HTTP Request Do-er, and you downscope your tests to your business logic exclusively.
Lots of people have different opinions on this. You're entitled to yours, but I think it's hasty to call it bad advice.
I'm not so worried about http.Client behaving incorrectly, but I am worried about the server I'm targeting behaving in a way that's different than my expectations. I'm also worried about getting cookie, post-data, url, etc. formats wrong. Even if they match what I put in my mock, they won't necessarily match what's out there in production. I'm worried that I'll handle download resumption wrong, for example.
If I have the capability to bring up the target server locally, that's what I will do. If I can't, but it's straightforward to create a work-alike, I'll bring that up in process. Only in the most dire of circumstances will I use mocks to test my code.
Think about it this way. What harm is done by using the real object, under the circumstances I described? Sure, if the dependency code breaks, my tests will too. Is that bad? Is there any point in my tests passing when my code won't actually work in production? OK, so maybe the tests will run slower? This matters in the limit, but often times you can use the production dependency without materially increasing test time. OK but what about the CI system using more resources? That will only be a problem of significant economic import if you're running at a scale like Google or Facebook.
My opinion is there are a few main valid objections to my approach. One is an argument about the size of the test code. In some cases, bringing up the production dependency will require more code than a mock. Sometimes a lot more code. My response is that we should make it easier to bring up the production dependency! But if we can't, I can see the case for a mock or a fake. The other counterargument is for when we really are interested in testing the sequence of interactions the code under test has with the production system. In that event, I guess I'm ok with using a mock, but I think most of the time we're more interested in testing state. We can also use things like testvalue injection to test behavior sequences, which is normally a lighter-weight approach, and in some cases can allow you to reproduce paths that you'd otherwise have trouble hitting, even with a mock.
It seems a bad advice also in my eyes.
I guess that you have never worked on big systems if for you it is normal to bring up all the production dependencies for an integration test.
For small projects it may make sense, although I would still prefer to write unit tests to catch exactly were the bug happens and to be independent from breakages in the external systems.
The correct approach is to rely heavily on unit tests that run continuously at every build and run the full integration testing suite at regular intervals or overnight.
In a quite big project on which I worked many years ago the full integration test suite took 11-12 hours to run, I would be curious how you would proceed in that case given that from what you write it seems that unit tests are pretty useless to you..
> I guess that you have never worked on big systems
I work on search and indexing systems at Google, and have done for more than a decade. I'm by no means the best, nor an authority. But I have spent some time living in this problem space, and this is my conclusion from my years of experience.
FWIW a single run of the tests for the system I work on now takes . . . I don't know. I would guess hours or maybe even a day of computer time? But, computers are cheap, and defects are expensive. So you split the test suite into bits small enough that the whole thing runs in ten minutes, and run them all in parallel. And, yes, there have been cases where I have had to debug the slow startup of a dependency so that I can make it fast enough to run in my test. But, that's what they pay me for, I enjoy that type of work, and it has ancillary benefits to the dependency when it needs to start up in production.
In the real world you don’t have pretty much unlimited resources.
If I remember correctly those tests I was speaking about were already running in parallel on 2 or 3 machines otherwise the total time would have been 2 or 3 times more. Obviously we couldn’t use hundreds of machines to make them run in 10 minutes.
In most of the places were I worked even getting a couple of new machines for production use would take months, I think if I requested even a couple of machines dedicated only for the integration testing I would have been laughed in the face.
So if you can run a full suite of integration tests in minutes on hundreds/thousands of machines then good for you, but don’t expect that every company has the resources for doing it.
And in those cases unit tests that run continuously are more useful than integration tests that run once a day.
> In most of the places were I worked even getting a couple of new machines for production use would take months
OK, well I guess this would be another caveat for my advice. If bringing up a production simulacra would take more resources than you have, then you might have to lean more on various types of doubles. This may be more often in environments that have insane prioritization of dev time vs. machine resources.
If I may ask, what was the bottleneck in your test suite? Did it saturate the cpu, ram, disk?
I don’t really remember.
The software was a kind of ORM but it was built to interface the java code with an object database, so it doesn’t make much sense to call it ORM.
Probably the main bottlenecks were cpu and disk if I have to guess.
I arrived very late in the project, at the point where we were going to migrate everything to oracle and we had to add oracle support to that behemoth.
My contention is that using real objects normally does not take materially more of any of these, and thus that most code is served best by being tested only by integration and regression tests. Where my contention is untrue, apply your best judgment, of course. :)
i wish i could give you 1 thousand points. isolated unit tests can have value in that the bugs they expose are more easily tracked down.
full-system testing, with an honest attempt to do fault injection, by definition exposes all the issues you might have in production. if you have a decent test suite you can honestly claim you can expect it to take client code correctly.
i don't know how thats true if you've only tested A against some stubbed out version of B that always returns success.
integrated testing is harder and requires more framework. its also kinda necessary, and if you only have the resources to do one or the other, the choice seems pretty clear.
i also heartily agree with your comment that since we're supposed to have frictionless and scalable deployment. there is no reason not to test this as part of integration testing and also exploit this function to make integration testing easier.
After reading the Martin Fowler article you linked, your comment makes more sense, but I think the terminological distinctions Fowler makes are never going to be widely adopted. People are going to keep using terms like "mock" and "stub" to broadly refer to any kind of test double, and it's disingenuous to assume otherwise.
> "People are going to keep using terms ... and it's disingenuous to assume otherwise."
I'm sorry, I'm making an argument about how people should write tests, not what terms they'll use to describe it. :) I promise you that there is no disingenuity going on in my use of certain terminology to describe how I think it's best to write tests.
The general theory is that, although you're focused on the code under test, if the underlying dependency is not behaving as expected, you want to know that. Mocks create an opportunity for your code's expectations about your dependencies' behavior to diverge from reality. So if at all possible, they should be avoided.
There are a few general ways to do this. The best way is to use the real underlying dependency. This should be the default.
If the dependency is too slow to use in the test (e.g. it has a long startup time, or is hosted remotely), the next best thing is to use a fake -- a local implementation of the dep that has the same behavior, but operates in memory on ephemeral data. The best libraries will provide these, and if they don't, you might be able to implement one and contribute it back. Even if you have to maintain it yourself, it often takes less code than describing mock behavior. The downside of fakes vs. the real implementation is that their behavior too can diverge. But this is not a net downside compared to mocks.
If you can't use a fake, the next best thing after that is a stub. See below.
Only if none of these options are viable should you use mocks.
1. Don't put your internal libraries in /pkg. /pkg has special GOPATH meaning as "compilation cache". Its not an actual name conflict, but why bother risking it.
2. This is just wrong:
> fmt.Printf is self-contained and doesn’t affect or depend on global state; in functional terms, it has something like referential transparency. So it is not a dependency. Obviously, f.Bar is a dependency. And, interestingly, log.Printf acts on a package-global logger object, it’s just obscured behind the free function Printf. So it, too, is a dependency.
stdout (and the buffer, and mutex on it) are exactly the same as the global log object. In fact, `log.Printf` is more or less just an alias to `fmt.Printf`[0]
3. I wish it had mentioned the functional options pattern in the part about constructors[1]
0: https://github.com/golang/go/blob/b77aad089176ecab971d3a72f0...
1: https://dave.cheney.net/2014/10/17/functional-options-for-fr...