Selecting Scientific Tools

This page started as a lab meeting discussion of how to choose among different options for scientific tools (mostly focused on programming).

Question Framing

  • When should I learn python/R/Stan/etc.?
  • How does {tool x} integrate into my workflow?
  • If I don’t learn {tool x}, what am I doing instead?
  • What are the benefits for knowing how to use {tool x}
  • What are the benefits for learning how to use {tool x}

Identify Alternatives

  • some possible scenarios
    • I have a specific research problem that doesn’t appear to solvable with my current toolset
    • Lots of people are using {tool x}, and I don’t want to be left behind
    • No one else is using {tool x}, and I want to be the leading expert in my lab/department/field
  • what are some other considerations beyond solving my specific problem
    • speed
    • generality
  • how do I find out about other options?
    • google +3
    • twitter +3
    • literature search
    • seminar talks, conferences (people tend to be a little more “in the weeds”, more receptive for engagement & dialogue)
    • workshops
  • what are the people around me using?
    • peers, colleagues, collaborators
    • similar enough level for learning/teaching

Evaluate the utility of knowing a new tool

properties of communities to consider:

  • communities to avoid (non-inclusive, wrong level)
  • asking about what communities are set up around tools
  • how diverse is the steering committee for the tool
  • what are the practices of the community?
  • If you are going to spend time in a community, will you be able to grow and take on leadership roles? (is that important for you)

What is the long-term trajectory of a tool:

  • is it actively being developed?
  • established history of developers
  • support for older versions

Evaluate the utility of learning a new tool

  • humans are habit-forming
    • defer learning new tools –> (decades later) “I’m too old to learn {tool x}”
    • always learning new tools –> never get around to using anything in practice
  • exposure to new communities builds your professional network
  • exposure to new ideas gives added context to what you already knew before
  • “learning tools” is also a skill! (which means you can practice it and get better)
  • “if you don’t use it, you lose it” -

Things to keep in mind

Decision making is difficult (but it is a skill you can practice and improve on!)

  • imperfect knowledge

    • no information source is incomplete
    • information sources aren’t completely tuned to your individual situation
  • outcomes are imperfectly measured

    • how useful was learning {tool x}? (no ideal way to measure this)
    • no measurement of what would have occurred with an alternative
  • limited time/attention

    • spending (more) effort to improve decisions can also be suboptimal
  • when to abandon bad decisions (sunk cost fallacy)

Tips

  • find good heuristics
    • be aware of biases (cognitive and otherwise)
    • reflect on past decisions (successful and unsuccessful)
  • engage in deliberate practice
    • identify appropriate goals
    • evaluate outcomes
    • get targeted feedback
    • repeat
  • embrace failure
    • “People don’t fail, processes do”
    • debug your processes (learn from mistakes, aim for long-term growth)
  • learn to fail fast
    • for large/complex projects - is {tool x} suitable long-term?
    • test whether the {tool x} addresses the most difficult aspect of the project
      • if this doesn’t work, it is better to know sooner!
  • exposure vs. learning
    • goal-oriented or not?