Difference between revisions of "Project and Design Documentation"

Revision as of 09:06, 20 September 2014 (view source) Casimono (talk | contribs) (→‎Features and Functions) ← Older edit		Revision as of 09:08, 20 September 2014 (view source) Casimono (talk | contribs) (→‎Features and Functions) Newer edit →
Line 332:		Line 332:
	format as we look to the future.		format as we look to the future.
	</li>		</li>
		+	</ul>
		+	</p>
		+	</dd>
		+	<dt><b>Graphs></b></dt>
		+	<dd>
		+	<p>Provide support for graphs, charts, etc., such as:
		+	Asset allocation pie chart, portfolio value vs. cost,
		+	ROI.  Graphs should be printable to printer.
		+	Graph generation should be fully integrated with reporting,
		+	both for data collection via queries, and for displayed
		+	output.
		+	</p>
		+	<p>
		+	<b>Status:</b>
		+	<ul>
		+	<li>Different graphing packages were evaluated,
		+	GUPPI.
		+	Guppi was chosen.  Considered & rejected were
		+	plotutils, gnumeric graphing code (Miguel says
		+	they'll replace gnumeric code with guppi.)
		+	Miguel's/Gnumeric requirements were:
		+	interactive plot editing -- each segment attributes
		+	totally settable/controllable -- drag/move callbacks
		+	when segments are click-dragged.
		+	</li>
		+	<li>Basic pie charts and bar charts are used in GnuCash
		+	reports.  There are problems: dates along the bottom
		+	of a bar-chart are not well-spaced/autoscaled.
		+	The over look is a bit klunky.
	</ul>		</ul>
	</p>		</p>

---

Revision as of 09:08, 20 September 2014

General

The people behind GnuCash aim to create a world-class GPL'ed Open Source Personal Financial Application for GNU/Linux and other Unix's.

This page documents the technical issues and development directives surrounding this project. It is a kind of an FAQ for developers and contributors, providing status, and guidance.

If you simply want to get a better idea of what GnuCash is and what it does, visit its GnuCash home page or browse the GnuCash Wiki page. The home page contains screen shots, news items, and mailing list archives.

Architectural Goals

There are some over-reaching design principles and philosophies that we intend to maintain. Some of these concepts and terms are introduced in this section.

Separation of GUI and Data

First, we must maintain a clean separation between the data structures and the GUI that manipulates them, along the lines of the Model-View-Controller paradigm.

Model 

Lists of accounts and the transactions in them can be thought of as a representation of financial data, a Model.

Controller 

The GUI that adds, modifies and deletes these should be thought of as a manipulator of the data, a Controller. Thus, the Motif or Gnome GUI's are merely two possible manipulators of the data; others, based on e.g. web/cgi-bin, Qt/KDE, emacs, Java applets or Java servlets ought to be possible.

View 

The View of the data is a subset or slice of the data described by the Model. The View may consist of only the transactions for the month of May, or only the account totals for certain accounts. The View is used in part to generate the reports and graphs, but it is also that which the Controller interacts with. Views are generated by queries to the data store.

GnuCash also needs to deal with multiple distributed datasources: stock quotations from the net or transaction confirmations from online banks and brokerage houses, or from more mundane sources, such as file imports, or merger of data from several users. Amongst these terms, the concept of a global Model-View is dated, and somewhat inappropriate. Rather, we need to be concerned about how data is represented in the local address space of the GUI, how the GUI manipulates it, how data is brought in and merged from external sources, and how that data is again output, whether to a file or a local or remote database.

Thus, the View essentially represents a local data cache of the data that is immediately present and being displayed, reported, and manipulated. The Model is the abstraction of that data that the GUI (the controller) can act on.

Financial Engine

In GnuCash, the Model is implemented via the Engine API,and the View is the data that is currently in the Engine. Thus, the Engine is a set of programming API's that the GUI (or a script, or even a clever command-line-addict) can use to manipulate the data.

Currently, the Engine provides basic accounting structures. These include:

Transactions 

consist of a set of splits or journal entries (JE's) whose values sum to zero. The transaction includes several date fields, a description, and a common-currency field, and a universal unique id (uuid/guid). It also provides hooks to store arbitrary data associated with the transaction (using a URL-based key-value tree).

Split 

(or journal entry) is an amount in the account on which it is drawn. Splits also store reconcile status, dates, a memo field, and also a key-value based hook for arbitrary data.

Accounts 

include a name, a type, a description field, and the type of commodity they store. In principle, the account consists of a list of splits.

Chart of Accounts 

is a hierarchical tree of accounts.

The Engine has a basic two-phase commit model, and a query mechanism for finding the data needed for reports and views. The goal of the two-phase commit and query model is allow the creation of multi-user server based backends, such as an SQL backend, and RPC client-server backend, or an XML-based HTTP/web backend. This design seems to work for the above-named backends.

The Engine currently handles only a basic set of data sources:

• It can import and merge in QIF's (actually, this function has been moved into the GUI, and is no longer part of the engine)

• It can read and write its own XML byte stream; This ability has been used to provide a multi-user client-server demo (which is currently broken).

• It can use a Postgress SQL database as a datastore, thereby enabling multi-user and auditing functions.

• It can talk, via RPC, to a gnucash server. (This code is 'alpha' and incomplete/broken).

• It can get stock quotes from the net (actually, this function is provided by a separate module, the Finance::Quote perl module.)

However, since the Engine is meant to be the interface between the GUI and the financial data, it is really intended to be able to do much more. In particular, it should be possible to create a peer-to-peer network model, where gnucash peers can synchronize data between themselves. The engine should also be expandable to handle other sources of data, such as OFX/IFX, the Open Trading Protocol, or the OMG CORBA General Ledger submission. In particular, it should be possible to use GnuCash not only to view data from these sources, but also to manipulate it and send it back.

Modularity, Extensibility and Customization

The above structure should lead us to view GnuCash not so much as a tightly integrated application, but rather as a loose confederation of component objects, libraries and interfaces. This has a number of advantages for both the developer and the user. For the developer, it allows parts to evolve semi-independently of one-another, and to be used in other, non-gnucash projects. For the user, a good extensibility allows the use of #arrangements": a way of broadly customizing the appearence and behaviour of gnucash, and then allowing users to very easily share these customizations with one another. Such arrangements might be collections of canned reports, e.g. for business or home user. Or they might be the menu contents: one menu arrangement is for beginners, another for 'power users', a third for business owners. The goal is that broad areas should be not only customizable, but it should be possible, even easy, to trade these customizations between users.

In order to facilitate the gluing together of these parts, as well as simplify the questions of customizability, change and rapid development, GnuCash makes use of the Scheme extension language (as implemented in the FSF Guile interpreter), to glue the pieces together. (Note that the engine interface is also available with Perl interfaces, thanks to a SWIG wrapper.

A Web Browser for Financial Data

More and more finacial data is moving onto the web. People shop on-line. They pay bills on-line. There are even some promising e-wallet systems (such as WebFunds). For GnuCash to be relevent in this on-line world, it must be able to interact with these systems. There are several steps that can be taken along this path. First, it must be possible to simply and transparently import financial data off the web. Click on a QIF file, mime-type application/x-qif? Gnucash sucks it in without burping. But, in a more distant future, can GnuCash originate transactions? It should be able to!

GnuCash can be and should be "The Sophisticated Financial Web Browser". All GnuCash reports and the help system are HTML-based. In fact, GnuCash has a built-in web browser that can view ordinary web sites. Interactive gnucash components such as the bar and pie charts, or the graphs and reports, can all be served up by a remote web server, as well as locally through the GnuCash application. This allows GnuCash to blur the line between web browser and financial application, and offer the best of both worlds.

Why is this a good thing? One can always have a pure web-based accounting solution (such as SQL-Ledger) that uses ordinary web browsers to view the financial data. But there are problems: dynamically-generated html and cgi-bins aren't as pretty or easy to use, or as fast, responsive or sophisticated, as what you can do with a custom client. That is, GnuCash can be more interactive, easier to user, slicker and more professional looking in style and presentation than a plain-old dynamic website. Financial ASP's simply cannot currently offer the kind of utility, flexibility, responsiveness and integration that GnuCash can offer.

In theory, the 'Java revolution' was supposed to provide this function, with downloadable Java applets providing the 'fancier- than-plain-html' interface. But a Java applet that is this sophisticated would also be large and slow to download: it offers no inherent advantages over native code. Another problem with Java is the inherent propreitary fragmentation: no two Java applets are alike: every site has thier own; there is no standardization.

In order to solve the 'no gui standardization between websites' problem, there are a number of businesses that have arisen to provide "financial aggregation". You sign up with these sites, and let them get all of your credit-card, bank and investment info, and they can provide a unified interface for your bank statements, with graphs and reports (all for a monthly fee). The existance of these services highlights a problem: the inability to aggregate the same data on the users desktop, in a simple, coherent fashion. Let us posit that GnuCash, with the right on-line interfaces, would be a natural for this. For the paranoid, this has the added advantage that no one business has access to all of your financial records.

To put it another way, GnuCash can provide a centralized, trusted store for financial data that no other application can provide. The convenience factor: if you have trading accounts at e*trade and charles schwab, then GnuCash can be the central place where you can oversee all of your investments. The trust factor: maybe you can trust your web bank. Maybe you can trust your web stock account. But can you trust a single web entity with *all* of your financial data? Someone who won't treat you as 'just another consumer' and sell your 'consumer profile' to anyone who cares to stalk (ahem, target) your financial activity (ahem, purchasing) patterns? Didn't think so. Of course, you

• could* use some proprietary financial software. Assuming, of course, that you trust it not to have any built-in covert channels:

nothing that might send back the make and model of your CPU and the last ten transactions 'home to mommy' for 'diagnostic purposes'. Because open source software, such as GnuCash, can be audited it can be trusted in ways that no proprietary software can be. In an increasingly net-connected world, the ability to build trust through audits will be increasingly important.

Looking Into the Future

Things like support for OFX, and small-business features are hopefully in the near term future of gnucash. Lets take a look at some more far-out possibilities.

E-Wallet 

An e-wallet, just like a real wallet, stores cash. You can use that cash to pay freinds or businesses. Just like a real wallet, an e-wallet should not charge you monthly fees. It should provide you with some measure of anonymity and privacy. An e-wallet should also do something no ordinary wallet can do: show you a monthly report of expenditures. GnuCash should get e-wallet features.

Authentication, Authorization and Access Control 

How does an on-line merchant know that you are who you say you are? How can you use GnuCash to authorize a bill payment? Sooner or later, smart-cards will provide the sort of authentication and access control that is only groped for with PGP/GPG, keyrings, x.509 certificates, and Microsoft Passport/Hailstorm. Of all the electronic things in the electronic world where access control and authentication are important, real money is the most so.

Bill Presentment, Bill Pay 

These magic keywords just mean that a merchant wants to send you a bill, and wants you to go to thier website to pay that bill (they want your eyeballs). Of course, for the user, it would be more convenient to have the bill show up on the desktop, inside of gnucash, and get authenticated and paid out from the desktop (by gnucash), as you would do with all of your other bills.

Requirements

Lets take a look at who the current and potential future users of GnuCash are, and how they might use it.

The classes:

• Basic Household Accounting/Budgeting
• Personal Stock Portfolio Management
• Personal On-line Trading (Day-Trading)
• On-line shopping and bill-pay
• Small Business Users

These different applications may use some of the same financial terminology, and hopefully might share a fair bit of code, but have quite different goals and requirements.

Basic Household Accounting and Budgeting 

Important properties of a personal finance system include:

• Approachable and usable by occasional users who are not knowledgeable about accounting.
• Ease of use and simplicity is critical.
• A reasonable selection of reports, graphs, charts, and tools for personal finance, such as mortgage calculations.
• Budgeting support needs to be targeted at native users.
• Interfaces to on-line banking, shopping, stock systems. Bank and credit-card statements should arrive 'automatically' and always be up-to-date.

Personal Portfolio Management 

• Support for management of stock portfolios that may involve considerable sophistication, since individuals commonly have retirement plans that hold mutual funds, stocks, options, bonds, and the like.
• Reporting infrastructure needs to simplify handling of tax issues (long-term vs. short-term capital gains/losses, cost-basis FIFO accounting, simplfied cost-basis spinoff/merger handling). Need on-line updates of prices, simple portfolio overview, ability to link to websites for additional research.
• Integration with on-line trading systems. This could save time typing.
• Basic Audit features. There's a problem with blindly allowing on-line data (prices, transactions) to enter GnuCash. It may not be clear where it came from, and even if the source is beleived to be 'reputable', there still may be factual errors in the data. Thus, there must be a way of audting newly-arrived (or even old) on-line data, and mark it as 'reconciled', i.e. manually reviewed and checked for accuracy. That is, incoming on-line data must be auditable, and audit trails should mark the history of on-line data import.

Small Business Needs 

• With a business system, it is likely that there will be users who use it eight hours a day, which puts the emphasis on efficiency of user interface rather than on its approachability to naive users.
• Business systems require network support, and the ability to support multiple simultaneous users.
• Some business users may want access to the system from an MS Windows 95/98/NT box. For these folks, a web-based interface could be just handy. Web interfaces are also nice a for ASP type deployment.
• Small businesses do not often have sophisticated investment portfolios; they instead need support for additional sophistication in such areas as:
  • Customer and Vendor Lists; Invoicing
  • Payroll (Batch processed and individual)
  • Inventory Control & Asset Management
  • Amortization Schedules, Depreciation
  • Shipping and Receiving
  • Accounts Receivable, Accounts Payable (A/R, A/P)
  • Credit Card Processing
• Support for calculations associated with accrual accounting.
• Ambitions for the future might include interfaces to online shopping carts, credit card clearing interfaces, and ERP systems.

Reconciling Those Needs 

A seemingly contradictory factor is that the kinds of sophistication that are required vary considerably. Consider:

• A home user does not generally require most of the sophistication of accrual accounting that is required by business enterprises. Thus, home users don't need much of the sophistication of an Accounts Receivable or Payable system, or the bizarre depreciation policies that crop up in Asset Management systems.
• On the other hand, home users are in increasing need of surprisingly sophisticated financial tools for managing stock portfolios, including:
  • Mutual funds
  • Retirement savings plans, with such identifiers as 401(k), IRA, Roth IRA, RRSP, or Insurance Annuities
  • Corporate stock purchase plans
  • Corporate option plans

Another set of contradictory requirements has to do with the back-end, and interfacing to other systems:

• Home users need a simple-to-install, simple-to-maintain system. This essentially rules out the use of SQL for the storage medium/back-end for home users. (That is, the current state of the art for SQL on Linux does not offer any simple, fool-proof management for data).
• By contrast, non-SQL systems for business use are almost unimaginable. SQL provides a high degree of data integrity and storage robustness, and also simplifies tremendously the import and export of data. Powerful SQL tools exist that can work magic in the hands of a good DB admin.

It may be that these will require completely different systems, and that GnuCash cannot be "all things to all people." This remains to be seen.

Features and Functions

This section reviews the current status of various features. Some of these are 'in process', some are 'almost done', some are 'completely done'. This section thus provides status on both where we've been, and where we're going.

Internationalization

All menus, markup and help-text should be internationalized, so that GnuCash may be usable in any country. This would include the printing of currency values in the local country conventions.

Current status: Essentially Done (?)

• All GUI messages currently use GNU gettext() for the message catalogs. Translations exist for English, British, French, Swedish, German, Japanese.
• Help pages available only in English and French.
• Monetary and string handling done through glibc. The latest glibc (2.2.3) is needed to get the correct functions.
• Yannick Le Ny <y-le-ny@ifrance.com> traduction en francais
• Most GUI input elements use the gtk text widget, and thus use the XIM input method in Asian locales. This allows e.g. Kanji, Katakana support. However, the register does not use XIM, and thus doesn't currently support the Asian languages. This needs fixing. (This may be done already??)

Reports

A variety of reports, including Net Worth, Balance Sheets, and Profit and Loss statements. These should be printable: that is, exportable as HTML as well as print-ready postscript. These should be easy to customize. Ideally, even novice users should be able to create custom reports.

The Report Generator should be a separate but "dockable" subsystem of the whole. That is, it should be possible to run the report generator in a stand-alone, read-only fashion without having to start up the main application. It should be possible to run reports nightly from a command-line and/or cron job. The GUI should remember what reports were run last time, and these should be re-run/redisplayed whenever the report generator is used.

One difficult aspect of reporting is designing a configurable interface, so that people can build custom reports. The New Reporting Infrastructure is seeking to build this up using Guile. Note there are several flavours of customization:

• Allow user to specify a custom logo (e.g. company logo/address) on every page (generically, having a header and footer for every page).
• Allowing user to modify report title/subtitle on the fly (and possibly add notes at the top or bottom of the report, e.g. to explain line items).
• Allow user to use the transaction query interface to pick the set of transactions that will make up the report.
• Memorize the report that was asked for, give that report a name, and in the future, allow that same exact report to be re-run. Allow user to edit this report properties at a later date. Note that the basic idea is similar to that of memorized queries, discussed below.

       Note that the customization info should be stored in a 
       Arrangements File (see below).

Generated reports should be exportable to other gnome systems (probably using bonobo). Reports should also be exportable to the Gnumeric spreadsheet (probably by writing out gnumeric file format). Export of CSV (comma separated values) and tab-delimited formats (for other spreadsheet import) would be good. Tables & etc. should be exportable to AbiWord, StarOffice, other word processors. (formats: docbook sgml? would then make convert to richtext, TeX easy.) Export should be as wysiwyg as possible.

Must be possible to e-mail reports (for example, invoices) to users. Suggest an evolution addressbok /mailer bonobo plugin.

Relationship to budgeting not clear ...

Stock portfolio tools should include a Cost Averaging report, Market Index report, Stock Option values, Estimation of capital gains tax liabilities.

Reports should be printable to printer (postscript/ Adobe Acrobat). <p> Status:

• A general reporting infrastructure was implemented in Perl, in the form of html-embedded perl (ePerl). However, this reporting mechanism was abandoned in part because ongoing build and install problems related to ePerl and swig. Also, since ePerl didn't participate in the interpreter even loop, the report generator had to run as a separate process, reading data via pipes. This was uglier than some folks liked.
• A general reporting infrastructure has been implemented in Scheme. Currently, there are a variety of reports for Profit/Loss, Balance Sheet, and portfolio valuation; none are particularly sophisticated.
• Done: Reports are displayed with the gtk-html widget. This widget provides postscript printing and Acrobat output.
• There is currently no way (no longer any way??) to generate reports from the command line ...
• While many reports have been implemented, there is no master list of what we should have. We should have ....
• The following technologies were rejected/unused mostly because they were too complex, didn't hang together technologies: SGML and Extensible Markup Language - XML. In the long run, these are preferable to HTML, since DSSSL tools such as Jade (James DSSSL Engine) can be used to convert to RTF, Postscript, etc. Add to this the consideration that XML is the basis for the Document Object Model, which is being integrated into many web-based applications, and we can see that XML is an increasingly significant format as we look to the future.

Graphs>

Provide support for graphs, charts, etc., such as: Asset allocation pie chart, portfolio value vs. cost, ROI. Graphs should be printable to printer. Graph generation should be fully integrated with reporting, both for data collection via queries, and for displayed output.

Status:

• Different graphing packages were evaluated, GUPPI. Guppi was chosen. Considered & rejected were plotutils, gnumeric graphing code (Miguel says they'll replace gnumeric code with guppi.) Miguel's/Gnumeric requirements were: interactive plot editing -- each segment attributes totally settable/controllable -- drag/move callbacks when segments are click-dragged.
• Basic pie charts and bar charts are used in GnuCash reports. There are problems: dates along the bottom of a bar-chart are not well-spaced/autoscaled. The over look is a bit klunky.

GnuCash Architecture

Decisions

Project Decisions

Design Decisions

Rules

Project Rules

Design Rules

Coding Rules