adding ecology pages

Author: butler-julie

EcologyLesson/.DS_Store

@@ -0,0 +1,89 @@
+---
+title: "Introduction"
+subtitle: "Data Organization in Spreadsheets for Ecologists"
+---
+
+## Overview
+* **Teaching:** 15 min
+* **Exercises:** 3 min
+* **Questions**
+	* What are basic principles for using spreadsheets for good data organization?
+* **Objectives**
+	* Describe best practices for organizing data so computers can make the best use of data sets.
+
+Good data organization is the foundation of your research project. Most researchers have data or do data entry in spreadsheets. Spreadsheet programs are very useful graphical interfaces for designing data tables and handling very basic data quality control functions.
+
+## Spreadsheet outline
+
+After this lesson, you will be able to:
+
+*bImplement best practices in data table formatting
+* Identify and address common formatting mistakes
+* Understand approaches for handling dates in spreadsheets
+* Utilize basic quality control features and data manipulation practices
+* Effectively export data from spreadsheet programs
+Overall good data practices
+
+Spreadsheets are good for data entry. Therefore we have a lot of data in spreadsheets. Much of your time as a researcher will be spent in this ‘data wrangling’ stage. It's not the most fun, but it's necessary. We'll teach you how to think about data organization and some practices for more effective data wrangling.
+
+## What this lesson will not teach you
+
+* How to do statistics in a spreadsheet
+* How to do plotting in a spreadsheet
+* How to write code in spreadsheet programs
+
+If you're looking to do this, a good reference is [Head First Excel](https://www.amazon.com/Head-First-Excel-learners-spreadsheets/dp/0596807694/), published by O’Reilly.
+
+## Why aren't we teaching data analysis in spreadsheets
+
+* Data analysis in spreadsheets usually requires a lot of manual work. If you want to change a parameter or run an analysis with a new data set, you usually have to redo everything by hand. (We do know that you can create macros, but see the next point.)
+
+* It is also difficult to track or reproduce statistical or plotting analyses done in spreadsheet programs when you want to go back to your work or someone asks for details of your analysis.
+
+
+## Spreadsheet programs
+
+Many spreadsheet programs are available. Since most participants utilize Excel as their primary spreadsheet program, this lesson will make use of Excel examples.
+
+Free spreadsheet programs that can also be used are LibreOffice Calc, and even Google Sheets.
+
+Commands may differ a bit between programs, but the general idea is the same.
+
+Spreadsheets encompass a lot of the things we need to be able to do as researchers. We can use them for:
+
+* Data entry
+* Organizing data
+* Subsetting and sorting data
+* Statistics
+* Plotting
+
+We do a lot of different operations in spreadsheets. What kind of operations do you do in spreadsheets? Which ones do you think spreadsheets are good for?
+
+## Problems with Spreadsheets
+
+Spreadsheets are good for data entry, but in reality we tend to use spreadsheet programs for much more than data entry. We use them to create data tables for publications, to generate summary statistics, and make figures.
+
+Generating tables for publications in a spreadsheet is not optimal - often, when formatting a data table for publication, we're reporting key summary statistics in a way that is not really meant to be read as data, and often involves special formatting (merging cells, creating borders, making it pretty). Cutting and pasting from a spreadsheet to a document software (like Word) can have unpredictable results. We advise you to create tables within these document software using the document's own table editing software.
+
+The latter two applications, generating statistics and figures, should be used with caution: because of the graphical, drag and drop nature of spreadsheet programs, it can be very difficult, if not impossible, to replicate your steps (much less retrace anyone else's), particularly if your stats or figures require you to do more complex calculations. Furthermore, in doing calculations in a spreadsheet, it's easy to accidentally apply a slightly different formula to multiple adjacent cells. When using a command-line based statistics program like R or SAS, it's practically impossible to apply a calculation to one observation in your data set but not another unless you're doing it on purpose.
+
+## Using Spreadsheets for Data Entry and Cleaning
+
+However, there are circumstances where you might want to use a spreadsheet program to produce “quick and dirty” calculations or figures, and data cleaning will help you use some of these features. Data cleaning also puts your data in a better format prior to importation into a statistical analysis program. We will show you how to use some features of spreadsheet programs to check your data quality along the way and produce preliminary summary statistics.
+
+In this lesson, we will assume that you are most likely using Excel as your primary spreadsheet program - there are others (gnumeric, Calc from OpenOffice), and their functionality is similar, but Excel seems to be the program most used by biologists and ecologists.
+
+In this lesson we're going to talk about:
+
+1. [Formatting data tables in spreadsheets](../formattingtables/index.qmd)
+2. [Formatting problems](../formattingproblems/index.qmd)
+3. [Dates as data](../datesasdata/index.qmd)
+4. [Quality control](../qualitycontrol/index.qmd)
+5. [Exporting data](../exporting/index.qmd)
+
+## Key Points
+* Good data organization is the foundation of any research project.
+
+Licensed under [CC-BY 4.0 2018–2022](https://datacarpentry.org/spreadsheet-ecology-lesson/00-intro/index.html) by [The Carpentries](https://carpentries.org/)
+
+Licensed under [CC-BY 4.0 2016–2018](https://datacarpentry.org/spreadsheet-ecology-lesson/00-intro/index.html) by [Data Carpentry](http://datacarpentry.org/)

EcologyLesson/Intro/index.html

@@ -0,0 +1,141 @@
+---
+title: "Dates as data"
+subtitle: "Data Organization in Spreadsheets for Ecologists"
+---
+
+## Overview
+* **Teaching:** 10 min
+* **Exercises:** 3 min
+* **Questions**
+	* What are good approaches for handling dates in spreadsheets?
+* **Objectives**
+	* Describe how dates are stored and formatted in spreadsheets.
+	* Describe the advantages of alternative date formatting in spreadsheets.
+	* Demonstrate best practices for entering dates in spreadsheets.
+
+Dates in spreadsheets can be a problem. For one thing, dates are stored in a single column. While this seems the most natural way to record dates, it actually is not best practice. A spreadsheet application will display the dates in a seemingly correct way (to a human observer) but how it actually handles and stores the dates may be problematic.
+
+In particular, please remember that DATE functions that are valid for a given spreadsheet program (be it LibreOffice Calc, Microsoft Excel, OpenOffice, Gnumeric, etc.) DATE functions are usually guaranteed to be compatible only within the same family of products. Most of the images of spreadsheets in this lesson come from Microsoft Excel, run on a Mac or on Windows. Regardless of your spreadsheet, if you will later need to export the data and need to conserve the timestamps, you are better off handling them using one of the solutions discussed below.
+
+One of the big problems with Excel is it can [turn things that aren’t dates into dates](https://nsaunders.wordpress.com/2012/10/22/gene-name-errors-and-excel-lessons-not-learned/), for example gene/protein names or identifiers like MAR1, DEC1, OCT4 will be changed to dates, and you cannot retreive the original name or identifier (except manually). So if you avoid the date format overall, it’s easier to work with these types of data. When you must work with dates, here is how to do it efficiently.
+
+## Exercise
+Challenge: pulling month, day and year out of dates
+
+* Let’s create a tab called `dates` in our data spreadsheet and copy the ‘plot 3’ table from the `2014` tab (that contains the problematic dates).
+* Let’s extract month, day and year from the dates in the `Date collected` column into new columns. For this we can use the following built-in Excel functions:
+
+```YEAR()```
+
+```MONTH()```
+
+```DAY()```
+
+(Make sure the new columns are formatted as a number and not as a date.)
+
+You can see that even though we expected the year to be 2014, the year is actually 2015. What happened here is that the field assistant who collected the data for year 2014 initially forgot to include their data for ‘plot 3’ in this dataset. They came back in 2015 to add the missing data into the dataset and entered the dates for ‘plot 3’ without the year. Excel automatically interpreted the year as 2015 - the year the data was entered into the spreadsheet and not the year the data was collected. Thereby, the spreadsheet program introduced an error in the dataset without the field assistant realising.
+
+## Exercise
+
+Challenge: pulling hour, minute and second out of the current time
+
+Current time and date are best retrieved using the functions `NOW()`, which returns the current date and time, and `TODAY()`, which returns the current date. The results will be formatted according to your computer’s settings.
+
+1. Extract the year, month and day from the current date and time string returned by the NOW() function.
+2. Calculate the current time using NOW()-TODAY().
+3. Extract the hour, minute and second from the current time using functions HOUR(), MINUTE() and SECOND().
+4. Press F9 to force the spreadsheet to recalculate the NOW() function, and check that it has been updated.
+
+## Preferred date format
+
+It is much safer to store dates with [YEAR, MONTH, DAY](https://datacarpentry.org/spreadsheet-ecology-lesson/03-dates-as-data/index.html#day) in separate columns or as [YEAR and DAY-OF-YEAR](https://datacarpentry.org/spreadsheet-ecology-lesson/03-dates-as-data/index.html#doy) in separate columns.
+
+Note: Excel is unable to parse dates from before 1899-12-31, and will thus leave these untouched. If you’re mixing historic data from before and after this date, Excel will translate only the post-1900 dates into its internal format, thus resulting in mixed data. If you’re working with historic data, be extremely careful with your dates!
+
+Excel also entertains a second date system, the 1904 date system, as the default in Excel for Macintosh. This system will assign a different serial number than the [1900 date system](https://support.microsoft.com/en-us/help/214330/differences-between-the-1900-and-the-1904-date-system-in-excel). Because of this, [dates must be checked for accuracy when exporting data from Excel](http://uc3.cdlib.org/2014/04/09/abandon-all-hope-ye-who-enter-dates-in-excel/) (look for dates that are ~4 years off).
+
+## Date formats in spreadsheets
+
+Spreadsheet programs have numerous “useful features” which allow them to handle dates in a variety of ways.
+
+![Many formats, many ambiguities](index4_files/5_excel_dates_1.jpg)
+
+But these “features” often allow ambiguity to creep into your data. Ideally, data should be as unambiguous as possible.
+
+## Dates stored as integers
+
+The first thing you need to know is that Excel stores dates as numbers - see the last column in the above figure. Essentially, it counts the days from a default of December 31, 1899, and thus stores July 2, 2014 as the serial number 41822.
+
+(But wait. That’s the default on my version of Excel. We’ll get into how this can introduce problems down the line later in this lesson. )
+
+This serial number thing can actually be useful in some circumstances. By using the above functions we can easily add days, months or years to a given date. Say you had a sampling plan where you needed to sample every thirty seven days. In another cell, you could type:
+
+```=B2+37```
+
+And it would return
+
+```8-Aug```
+
+because it understands the date as a number `41822`, and `41822 + 37 = 41859` which Excel interprets as August 8, 2014. It retains the format (for the most part) of the cell that is being operated upon, (unless you did some sort of formatting to the cell before, and then all bets are off). Month and year rollovers are internally tracked and applied.
+
+**Note** Adding years and months and days is slightly trickier because we need to make sure that we are adding the amount to the correct entity.
+
+* First we extract the single entities (day, month or year)
+* We can then add values to do that
+* Finally the complete date string is reconstructed using the DATE() function.
+
+As for dates, times are handled in a similar way; seconds can be directly added but to add hour and minutes we need to make sure that we are adding the quantities to the correct entities.
+
+Which brings us to the many different ways Excel provides in how it displays dates. If you refer to the figure above, you’ll see that there are many ways that ambiguity creeps into your data depending on the format you chose when you enter your data, and if you’re not fully aware of which format you’re using, you can end up actually entering your data in a way that Excel will badly misinterpret and you will end up with errors in your data that will be extremely difficult to track down and troubleshoot.
+
+### Exercise
+What happens to the `dates` in the dates tab of our workbook if we save this sheet in Excel (in `csv` format) and then open the file in a plain text editor (like TextEdit or Notepad)? What happens to the dates if we then open the `csv` file in Excel?
+
+#### Note
+You will notice that when exporting into a text-based format (such as CSV), Excel will export its internal date integer instead of a useful value (that is, the dates will be represented as integer numbers). This can potentially lead to problems if you use other software to manipulate the file.
+
+## Advantages of Alternative Date Formatting
+
+Storing dates as YEAR, MONTH, DAY
+
+Storing dates in YEAR, MONTH, DAY format helps remove this ambiguity. Let’s look at this issue a bit closer.
+
+For instance this is a spreadsheet representing insect counts that were taken every few days over the summer, and things went something like this:
+
+![So, so ambiguous, it's even confusing Excel](index4_files/6_excel_dates_2.jpg)
+
+If Excel was to be believed, this person had been collecting bugs **in the future**. Now, we have no doubt this person is highly capable, but I believe time travel was beyond even their grasp.
+
+Entering dates in one cell is helpful but due to the fact that the spreadsheet programs may interpret and save the data in different ways (doing that somewhat behind the scenes), there is a better practice.
+
+In dealing with dates in spreadsheets, separate date data into separate fields (day, month, year), which will eliminate any chance of ambiguity.
+
+## Storing dates as YEAR, DAY-OF-YEAR
+
+There is also another option. You can also store dates as year and day of year (DOY). Why? Because depending on your question, this might be what’s useful to you, and there is practically no possibility for ambiguity creeping in.
+
+Statistical models often incorporate year as a factor, or a categorical variable, rather than a numeric variable, to account for year-to-year variation, and DOY can be used to measure the passage of time within a year.
+
+So, can you convert all your dates into DOY format? Well, in Excel, here’s a useful guide:
+
+![Kill that ambiguity before it bites you!](index4_files/7_excel_dates_3.jpg)
+
+## Storing dates as a single string
+
+Another alternative could be to convert the date string into a single string using the `YYYYMMDD` format. For example the date `March 24, 2015` would become `20150324`. This option also works for datetimes using the `YYYYMMDDhhmmss` format. So the datetime `March 24, 2015 17:25:35` would become `20150324172535`, where:
+
+* `YYYY`: the full year, i.e. 2015
+* `MM`: the month, i.e. 03
+* `DD`: the day of month, i.e. 24
+* `hh`: hour of day, i.e. 17
+* `mm`: minutes, i.e. 25
+* `ss`: seconds, i.e. 35
+
+Such strings will be correctly sorted in ascending or descending order, and by knowing the format they can then be correctly processed by the receiving software.
+
+## Key Points
+* Treating dates as multiple pieces of data rather than one makes them easier to handle.
+
+Licensed under [CC-BY 4.0 2018–2022](https://datacarpentry.org/spreadsheet-ecology-lesson/00-intro/index.html) by [The Carpentries](https://carpentries.org/)
+
+Licensed under [CC-BY 4.0 2016–2018](https://datacarpentry.org/spreadsheet-ecology-lesson/00-intro/index.html) by [Data Carpentry](http://datacarpentry.org/)